Once again, your surface-level reading (or quote lifting, as the case may be) does no one, including yourself any good. Here you can find a reprinting of his statement, in context. When read in context, the quote doesn't mean what you want it to mean.
Notice how he sets up the plain meaning here: "If there had been a desire on the part of any single State to withdraw from the compact at any time while the number of States was limited to the original thirteen, I do not suppose there would have been any to contest the right, no matter how much the determination might have been regretted."
Grant admits that while an informal secession may have been successful in the early days of the union, that door closed when the union admitted more states, and conclusively locked with the addition of the Louisiana Purchase.
But the times had changed, the circumstances had changed. The casual understandings of the relationships were no longer practical. Grant understood this. "Secession was illogical as well as impracticable; it was revolution."
"The fathers themselves would have been the first to declare that their prerogatives were not irrevocable. They would surely have resisted secession could they have lived to see the shape it assumed."
Reading is fundamental...;-)
Additionally:
“Let us, then, consider all attempts to weaken this Union, by maintaining that each state is separately and individually independent, as a species of political heresy, which can never benefit us, but may bring on us the most serious distresses.”
Charles Cotesworth Pinckney of South Carolina, 1740-1824, Revolutionary War soldier, one of the authors of the US Constitution in 1787, speaking at the South Carolina Ratifying Convention in 1788.