It seems to me that there were two contemporaneous US presidents who believed that there was no specific enumerated prohibition against secession.
It seems to me that one of those presidents said that he was powerless to stop the various secessions, but he did recognize that it was his duty and that he had specific authorizations and powers to deal with law breaking insurrectionists.
When they committed acts of violence and war against the United States he was honor-bound to respond. It wasn’t so much about what the rebels did as it was how they did it...
a practice subversive of all the principles of order and regular government; a practice which leads more directly to public convulsions, and the ruin of popular governments, than any other which has yet been displayed among us.
He wasn't saying it couldn't be done, but he did give the impression that the founders weren't any too keen on the idea.