Rhetoric aside, follow the money. It’s nearly always about power and money——“Any people, anywhere, being inclined and having the power, have the right to rise up and shake off the existing government, and form a new one that suits them better. This is a most valuable, a most sacred right, a right which we hope and believe is to liberate the world”.
Abraham Lincoln – U.S. Congress, 1847
A little over 10 years later after the South attempted precisely that, Lincoln, when asked, “Why not let the South go in peace”? replied; “I can’t let them go. Who would pay for the government”? “And, what then will become of my tariff”?
Abraham Lincoln to Virginia Compromise Delegation March 1861
That is one of those questionable Lincoln quotations.
If the slave states didn't want high tariffs, they could have stayed in Congress and defeated or moderated them.
Maybe it is all about money as you say, but in those days, slaves and slavery were a pretty big investment in some states -- big enough to secede to protect and big enough to fight for.