Absolutely correct. The genteel folk in Charleston didn't want noisy locomotives disturbing the tranquil downtown area. So any cargo had to be transferred from the rail cars, carried through town, and reloaded at the dock. Other cities built rail lines right to the port and passed the once thriving port of Charleston went into decline.
I think the decline of Charleston had more to do with a Federal invasion and destruction of capital and economic activity that they did than it did with anything Charleston did in mismanaging their businesses.