You focus is too narrow. Read what Madison wrote again. For the first time.
“The compact can only be dissolved by the consent of the other parties, or by usurpations or abuses of power justly having that effect.”
Beyond that, I can't believe that Madison (though he didn't sign the DOI) didn't believe that just powers are derived from the “consent of the governed.”
Almost all the quotes from Madison expressing the opinion that states could not unilaterally secede are from the 18-teens and 20s. This was of course long after ratification of the constitution and says nothing about what the parties - that would be the states - agreed to at the time.
The things Madison said about the powers of the federal government being severely limited and that of the states in no way forfeited or impaired he said before passage of the constitution. That is obviously what the states actually agreed to not his later interpretations.