I wouldn't consider either Madison or Adams bad presidents. For that era, though they weren't necessarily standouts among the great presidents (Washington, Jefferson, Monroe).
But your theory sure holds once college-educated presidents became the norm.
I get a particular chuckle over Columbia-- BO was the only one to finish; even the two Roosevelts had the sense to withdraw from that den of pinheads!
Well, IIRC, Adams and Madison were both one-termers. Adams, again, IIRC, gave us Chief Justice Marshall, who removed any doubt there might have been about the supreme complete and final power of the SCOTUS. He also gave us the alien and sedition acts, which are historically ignominious. Madison as president flip flopped on the national bank (was against it before he was for it) and flip flopped on standing army (against it, then for it) and I would say having the White House burned to the ground during your term in office is not exactly a benchmark of excellence :-)
Adams was a bad president because he was a big gubmint Federalist. Madison was simply a weak executive. h