Having four times the population and having the war fought in other people's land gives you quite an advantage in that regard.
An advantage to be sure. But that doesn't explain why Union soldiers didn't abandon the Union cause in droves when their enlistments ran out in the spring and summer of 1864. They did not, in overwhelming numbers.