Depends on which State they were citizens of when the call came. I don't begrudge a (former) Virginian living in Colorado for serving in the Union army. Longstreet instructed a sergeant and some enlisted men who were still on federal service in New Mexico to stay at their posts, when they asked him about traveling home to Virginia. He reasoned that he'd resigned his commission with a forward effective date (I suppose that's what Wlat is tootling about, but I'd have to see the documents and would be very unhappy to find out he was fooling with us), and so he himself, Longstreet, was released. But the Virginians in the rank were still under oath.
It would be harder to say about those selfsame Virginians if their State had indeed left the Union. I'd have to re-read Longstreet's memoir on that point. Could the Union bind a man to service against his own People under an oath? We never required it of the Nisei, but we sure did of the German-Americans in WW1 -- under pain of this, that, and the other. And that's how Longstreet seemed to see it.