Then why did southern leaders say it was?
The Southern states said the Northern states had violated the fugitive slave clause of the constitution - which they clearly had.
This was unconstitutional. This gave them a great argument for saying (correctly) that the Northern states had violated the compact between the states.
No matter how much they hated it, unequal treatment due to tariffs and unequal federal expenditures was not unconstitutional. Violating the fugitive slave clause of the constitution WAS unconstitutional.
If it was “all about slavery”, why did the North offer and why did the South reject explicit protection of slavery by constitutional amendment?.....ie the Corwin Amendment.