The growers sold to factors. Then the product was out of their hands. It would be great if farmers and manufacturers could sell directly to the end-user and cut out all the transportation, warehousing, sales commissions, and other costs involved in getting their product to the end-user but that’s a bit of a fantasy. Then again your whole view of the antebellum South is a complete fantasy.
Some well-off Southerners complained about New York and felt that they were being cheated, but the statistics used to support that view could easily be used to point out that they were dependent on Northerners providing commercial services that Southerners didn't provide themselves.
Others, though, wanted to stay agricultural, and wouldn't have had much problem with the British providing the same services that Northerners did. Still others didn't have the same grievances and pointed out that Southerners already were involved in shipping, finance, and other aspects of commerce and were already competiting with Northerners and those on the other side of the Atlantic.