You nailed it. Here's the logic: if you are against the idea of the government dictating terms of trade, you are necessarily in favor of slavery. We must empower the government to protect our freedoms.
The government isn't supposed to protect our freedom?
You read the federalist papers. Just what do you think its supposed to do?
"Fair Traders" want to restrict trade. If slavery is the denial of freedom, then denying some one the freedom to trade as they see fit, is a form of slavery. To the extent that the "Fair Traders" succeed, is the same extent they have made slaves of their fellow Americans.