Not so much of an issue if you are unloading goods and reloading up with Cargo. Especially at a 13% tariff rate as opposed to a 45% tariff rate.
Chopping off all the protectionist laws imposed by the North would have transformed trade and which ports would suddenly see massive growth.
Things like that don't always happen -- certainly not overnight. Colonies that become free of the mother country and its regulations often don't have the resources to make good once they became independent. Also, slaveowners' desire for a highly controlled society cut into industry and development and that wouldn't change with a drop in the tariff.