The founding fathers were well aware of Adam Smiths fairly tales. They rejected Smith. Thank God they did. Because the nation they built behind a protectionist wall was awesome.
The tariff enacted was a REVENUE tariff not a protectionist tariff. Recall the Nullification crisis when a real protectionist tariff was enacted.
Free trade would have clearly benefited the slave owners since it the products they imported were the ones affected most by the tariff. This had been the case from the beginning but the government had to have revenue.