Your point is reasonable. Not all slaveowners fit the Uncle Tom's Cabin stereotype. It is one thing to attempt to understand the slaveowner mentality. It is another thing altogether to try and justify it.
Justification they needed before Deity, and no doubt that depended on their treatment of their slaves, and whether it comported with Leviticus.
As for their politics, they needed no justification, since they had on their side the independence of the country, the law, and, through the ratification conventions and the Constitution, the consent of their neighbors -- not to put too fine a point on it, the consent of their Northern neighbors, who had ratified the Constitution at the same time they did.