Just a slight overestimation on our part.
and you point out that one southerner couldn't whip three Yankees, either.
Incorrect. We could and did for four years. Since deciding factors occur at the margin, the excess beyond 3 yankees for every southerner made the final determination. Normally the guy who loses the most men also loses the war. Y'all lost 100,000 more men than we did yet managed to squeek out a win - albeit a difficult one at that - for no other reason than that you were more populous.
Home-field advantage that the South got a taste of when they went to Gettysburg. Plus our best general defected so it's not the easiest thing to do to find a good general like Grant and Sherman.