Compare this with the approximately 3,890,000 slaves that show up in the 1860 census from the Southern and border states. The border states are defined as Maryland, Kentucky, and Missouri. The CSA "claim" Ky and Mo as their own, so that leaves Maryland. Maryland had over 87,000 slaves and nearly seceded.
So you compare fewer than 100,000 (most in one state) versus 3.8 million.
For the South, slavery was very much the issue with regard to secession. It is in every secession document and was on the tongue of every secession leader. The South wanted political parity, at least in the Senate, so it could preserve its "peculiar institution." When it was apparent that the slave-holding states would eventually lose parity because "free" states were beginning to enter the Union on the Plains and in the West, the South bolted. Notions of "States rights" and "State sovereignty" are just blowing smoke.