"I, Robert E. Lee, appointed a Lieutenant Colonel of the Second Regt. of Cavalry in the Army of the United States, do solemnly swear, or affirm, that I will bear true allegiance to the United States of America, and that I will serve them honestly and faithfully against all their enemies or opposers whatsoever; and observe and obey the orders of the President of the United States, and the orders of the Officers appointed over me, according to the Rules and Articles for the government of the Armies of the United States."
Note the bolded area. Which state do you follow when some of the states you've swore loyalty to break with the others. The oath as written recognizes a confederation of independent states, not a single nation. The US Army was jointly supported by the sovereign states.
RFL did exactly what he should have done, kept his oath to HIS state since it split from the others. At the time it split he ceased being a citizen of those United States because Virginia was no longer one of those states.
The Supreme Court ruled that none of the acts of secession had any force in law.
None of the rebels stopped being citizens.
Walt