The manned orbiter needn't be huge like the Space Shuttle. It would carry just personnel. Their luggage would be on the freighters, probably launched first.
An SSTO won't alter that equation very much -- you still have to launch and then land an aircraft. If anything, an SSTO carrying capacity will be far less than a Shuttle due to mass fraction limitations.
The real issue is getting stuff into space cheaply and easily. The Shuttle is expensive primarily because of how much manpower is required to prepare it for launch. Expendable boosters tend to cost far less per launch, but they're also far less flexible.
The main question for an SSTO is whether it can reliably turn around in 12 hours to perform the same mission. As with the Shuttle, the major issues affecting rapid turnaround are maintenance and durability of the vehicle structure and thermal protection systems; and the reliability of the propulsion systems.
If one or both of these turn out to be finicky -- as they likely will be due to their extreme performance requirements -- then an SSTO will not provide rapid turnaround.
In addition, integration of payloads with the vehicle tends also to be time-consuming.