I think his men were loyal to him because he won far more often than he lost.
Nothing could be further from the truth. If Virginia had stayed with the Federal government (which they very well could have done if a few votes had changed sides) Lee would have gladly lead the "War of Northern Aggression" against the Rebel states.
Lee fought for Virginia, not "The South".