Not at the time. He only said that in the 1820s.....he never claimed the ratifications were defective because states reserved the right to secede at the time.
In any event, he was not one of the parties to the compact. The states were and they made their intent quite clear at the time of ratification.
Read his letter to Alexander Hamilton on New York's conditional ratification. "My opinion is that a reservation of a right to withdraw if amendments be not decided on under the form of the Constitution within a certain time, is a conditional ratification, that it does not make N. York a member of the New Union, and consequently that she could not be received on that plan. Compacts must be reciprocal, this principle would not in such a case be preserved. The Constitution requires an adoption in toto, and for ever. It has been so adopted by the other States. An adoption for a limited time would be as defective as an adoption of some of the articles only. In short any condition whatever must viciate the ratification."
More than any other single man, Madison is responsible for the content of the Constitution and it's subsequent ratification.
...he never claimed the ratifications were defective because states reserved the right to secede at the time.
Did you not read the letter he wrote to Hamilton? I provided a link.