It was John Adams who opined that the U.S. Constitution was inadequate to the governance of any other than a moral people, and—while Adams's point might be debatable—it at least suggests that perhaps morality is a more legitimate distinction to be considered when considering separation—not the arbitrary consideration of what might be the color of one's skin...
You cannot judge the people of the 19th century with 21st century knowledge, hindsight and sense of morality. That applies to both Confederate and Union citizens of the 19th century.