He was loyal to his state. That meant a lot more back then.
Great point. Most people forget that while Lee was loyal to the Union at the start - he simply could not have turned against his own Virginia citizens like a traitor.