Madison was on all sides of the issue at some point in his life but was in agreement with St. George Tucker, Thomas Jefferson, George Mason, and many others in later years.
In what way do you mean?
I think you are confusing Madison with Jefferson. At various times in his life, nearly any faction today could find a Jefferson quote to back up their arguments. Madison, on the other hand was to my knowledge pretty consistent throughout his life.
but [Madison] was in agreement with St. George Tucker, Thomas Jefferson, George Mason, and many others in later years.
Well, I don't know what you mean by 'later in life' but here is an excerpt from Madison's letter to Daniel Webster in 1833 (3 years before he died) where he shoots down any constitutional basis for 'secession at will.' I'd say Madison was diametrically opposed to any idea of unilateral secession.
I return my thanks for the copy of your late very powerful Speech in the Senate of the United S. It crushes "nullification" and must hasten the abandonment of "Secession." But this dodges the blow by confounding the claim to secede at will, with the right of seceding from intolerable oppression. The former answers itself, being a violation, without cause, of a faith solemnly pledged. The latter is another name only for revolution, about which there is no theoretic controversy.