I don't see where that opinion trumps the opinion of those who believed that secession without the consent of the states was illegal. Madison certainly believed so.
Madison was on all sides of the issue at some point in his life but was in agreement with St. George Tucker, Thomas Jefferson, George Mason, and many others in later years.
And BTW:
The book I cited above was THE law textbook of the era and not disputed by anyone for quite some years.