If Southerners main concern was the new tariff they would have stayed in the union to keep tariff rates low.
And of course, most of the eventual rise in tariffs was accepted as a way of paying for the war.
But the point, I think, was more that if you were too poor to own slaves, you were too poor to have many foreign manufactured goods -- let alone luxury items -- and too poor for tariffs to make much of a difference to your way of life.
Backwoodsmen and poor whites and even ordinary farmers weren't that agitated about tariff rates.
It was people who were already relatively well to do who worried about tariffs and they worried far, far more about abolitionists and slave uprisings.