No one can successfully run a shipping business if the ships are required to arrive with empty holds, especially when their protected competition does not have to abide by such rules.
It is inherently a tilted playing field.
What if the ships stopped first at New York or Boston or Philadelphia to offload imports then sailed south to load up with cotton exports? That would limit the time spent in ballast and it the most likely explanation given the comparatively small amount of imports when compared with exports for southern ports.