I think that Madison is more explicit here:
"The conduct of S. Carolina has called forth not only the question of nullification; but the more formidable one of secession. It is asked whether a State by resuming the sovereign form in which it entered the Union, may not of right withdraw from it at will. As this is a simple question whether a State, more that an individual, has a right to violate its engagements, it would seem that it might be safely left to answer itself. But the countenance given to the claim shows that it cannot be so lightly dismissed. The natural feelings which laudably attach the people composing a State, to its authority and importance, are at present too much excited by the unnatural feelings, with which they have been inspired against their brethren of other States, not to expose them, to the danger of being misled into erroneous views of the nature of the Union and the interest they have in it. One thing at least seems to be too clear to be questioned; that whilst a State remains within the Union it cannot withdraw its citizens from the operation of the Constitution laws of the Union. In the event of an actual secession without the Consent of the Co-States, the course to be pursued by these involves questions painful in the discussion of them. God grant that the menacing appearances, which obtruded it may not be followed by positive occurrences requiring the more painful task of deciding them!"
And here Madison gets more to the point:
"An inference from the doctrine that a single State has the right to secede at will from the rest is that the rest would have an equal right to secede from it; in other words, to turn it, against its will, out of its union with them. Such a doctrine would not, till of late been palatable anywhere, and nowhere less so than where it is now most contended for."
And in his letter to Danial Webster, Madison minces no words:
"I return my thanks for the copy of your late very powerful Speech in the Senate of the United S. It crushes "nullification" and must hasten the abandonment of "Secession." But this dodges the blow by confounding the claim to secede at will, with the right of seceding from intolerable oppression. The former answers itself, being a violation, without cause, of a faith solemnly pledged. The latter is another name only for revolution, about which there is no theoretic controversy."