Under the chivalrous rules of war honored at the time, it was thought that Dolly would face less risk even if captured by the British whereas James would be a valuable war trophy. The president, of course, was against her remaining behind, but Dolly had other ideas. To suggest James was somehow cowardly is dead wrong.
I also think your statement that the whole Constitutional system was coming unglued when Andrew Jackson became president is a gross exaggeration. Jackson is considered one of our great presidents because he took some decisive action on things like killing the Bank of the United States. His immediate predecessor (John Q. Adams) was a nice fellow, but only a so-so president. The guy before that (James Monroe) was truly one of our greatest presidents because he ended European colonization of the Americas and the rest of the world actually respected the Monroe Doctrine up until the Bay of Pigs fiasco with a couple of minor exceptions.
The exceptions ended in disaster for the would-be colonizers. Just ask Emperor Maximilian how that French colonization of Mexico during our civil war turned out.
"Hence, it clearly appears, that the same advantage which a republic has over a democracy, in controlling the effects of faction, is enjoyed by a large over a small republic, -- is enjoyed by the Union over the States composing it. ...The influence of factious leaders may kindle a flame within their particular States, but will be unable to spread a general conflagration through the other States."
That's why I call the Federalist Papers a monument to error. Almost nothing in them turned out to be true. The closest thing to the truth in the Federalist Papers came from Hamilton, because he was much more open and clear about what he wanted--a strong national government, the obliteration of states' rights--and he understood that the Constitution would deliver on his wishes. Madison seemingly lived in a dream world.