As for James Madison, he was handilly re-elected in 1812, right at the start of the war. The Federalists ended up getting blamed for the burning of Washington two years later due to their tepid support for the war and never ran a candidate for national office again. Far from being a weak executive, Madison did a yeoman's job of uniting the country and ensured we had great presidents for the next 20 years after he left office.
Madison’s the one who fled the White House while DC burned. He gets the credit for that. As for the Federalists, didn’t they basically just morph into the Whig Party, and then eventually the Republican party?