The South's reasons for leaving is immaterial to the North's refusal to allow it for any reason.
The Salient question is this; Do people have a right to leave?
Our founding document, the Declaration of Independence, says "yes."
The South's reason for leaving is the heart of the discussion. The South's reason for starting the war were, as you yourself have said, were pretty stupid.