Too bad the southern secession didn’t bother to use that as the reason for their action.
Some did, though not directly. They complained about an excess of Federal power.
Here are some excellent articles (from a black writer) on the legality of secession:
http://walterewilliams.com/secession-or-nullification/
http://www.newsbusters.org/blogs/nb/walter-e-williams/2012/12/03/walter-e-williams-column-secession-craze-isnt-crazy-parting
http://www.wnd.com/2012/11/secession-its-constitutional/
http://walterewilliams.com/the-real-lincoln/
It would not have mattered if they had. Allowing them to get beyond the economic control of Washington D.C. and the New York "Robber Baron" power brokers, was not going to happen. One way or another, there would have been a war to stop them, because Washington D.C. and it's New York Crony Capitalism partners will never let go of a significant amount of money.
The South at the time earned 3/4ths of all Export Revenue for the United States. Too much money for the establishment to let go of peaceably.