Some buildings are constructed such that every 10-12' of exterior wall is supported by the floor below which is in turn supported by the wall below that, etc. The WTC was not constructed in such fashion. The walls themselves were pretty solid, and the floor trusses were fastened to them. The wall columns did rely upon the floor trusses to provide lateral stability, but the gravitational loads of the wall columns did not rest upon the floor trusses. Indeed, for a building of that height, putting the weight of the upper floors on the lower floors would have been absurd.
The wtc main support was provided by the central shaft, which in essence was a giant sized pillar (Like a tree trunk, but a little more complicated than that). Each slab also had "walls" which ran all the way to the exterior (in a cross shape from the center like you say. The pillars gave support and stability in between those for the floor above.
I think we are on the same page essentually. There is a blueprint around somewhere which shows how the towers were built.