There were plenty of southerners who remained loyal.
They did remain loyal - to their states. Most people at the time considered their state to be their country. Most had never been far from home.