It's worth noting that one of the factors in the Federalist collapse was the Hartford Convention, where the idea of secession was merely discussed and rejected, appearing nowhere in the final report of the convention. It was enough, though, for the south to loudly proclaim them as traitors. Here's what one Virginia newspaper said about it in 1814:
"No man, no association of men, no State nor set of States has a right to withdraw itself from this Union, of its own accord. The same power which knit us together can only unknit. The same formality which forged the links of the Union is necessary to dissolve it. The majority of States which form the Union must consent to the withdrawal of any one branch of it. Until that consent has been obtained, any attempt to dissolve the Union, or obstruct the efficiency of its constitutional laws, is treason-treason to all intents and purposes."
And for President Madison to move troop units stationed on the border with Canada back to Albany, NY, so as to be ready in case needed, if New Englanders declared secession.