He wasn't all that consistent at a young age either. Madison fluctuated between a moderate small-f federalist and a Jeffersonian free trader and nullifier over the course of just a couple years in the 1790's. He was a smart man and he had many valuable things to say, but he's simply unreliable for an unopposed authoritative opinion on many matters because for every letter to Daniel Webster that somebody else quotes I can quote the Virginia and Kentucky Resolves.
The bellicose attitude toward France of President John Adams,' administration alarmed Madison. The XYZ AFFAIR brought the United States and France close to war. During the subsequent turmoil in the United States, the administration won passage of the Alien and Sedition Acts, which Madison believed severely threatened free government. In protest he drafted the Virginia Resolutions of 1798 and a report defending them in 1800. These papers stated most fully Madison's concern to protect states' rights, but he advocated neither nullification nor secession, as John C. CALHOUN and others later asserted. Rather, the resolutions and report represent an important chapter in an evolving constitutional doctrine to defend civil liberties against encroachments by the federal government.
Happily retired to his Virginia farm, Madison practiced scientific agriculture, helped Jefferson found the University of Virginia, advised Monroe on foreign policy, arranged his papers for posthumous publication, and maintained a wide correspondence. He returned officially to public life only to take part in the Virginia constitutional convention of 1829, where he sought both to diminish the power of Tidewater slave owners and to extend the franchise. His compromise efforts fell before pressure from proslavery forces to preserve their dominance. Nationally, Madison wrote in support of a mildly protective tariff, the National Bank, and, most importantly, the power of the union against nullification. He stoutly denied that he had advocated nullification in the Virginia Resolutions of 1798. In fact, his whole career and his most profound political thought rested on securing for the United States the benefits of union. http://www.geocities.com/bkestory/madison.html
In fact, Madison was horrified that Calhoun and others would bastardize Madison's words to promote disunion.