To be fair, three states cited "slavery". Most didn't say anything at all, and the very most important state (Virginia) said they seceded because the government had turned tyrannical in it's efforts to force it's sister states into submission.
But I see it as a constant assertion from people making your argument that 3 states who said that, speak for 11 states that did not.
In a era of substantial property requirements to vote, slave owners had the upper hand politically. In effect, non-slave holders were mostly disenfranchised.
That is silly. The men who individually owned land in the South greatly outnumbered the wealthy plantation owners. You didn't get more votes for owning more land. It was still one vote per man.
The class aspect of secession is well-established. Oddly, in that respect you decline an economic interpretation, but it was widely commented on at the time, with the war being described as "a rich man's war but a poor man's fight." Smaller property holders, even non-slave owners, usually tended to have a major stake in the preservation of slavery because it was the foundation of the Southern economy and defined the Southern way of life.
The two major exceptions are notable, being the hill country of Alabama and West Virginia. In both cases, the poor quality of the land made it unsuitable for plantation agriculture, which mostly kept slavery out. They were mostly hostile to secession.