If you read the Declaration of Independence, you'll find the rebellious states identified the King's interference with slavery as justification for dissolving the political bands.
Those states included New Hampshire, New York, New Jersey, Massachusetts, Rhode Island, Pennsylvania, Connecticut, Delaware, and Maryland.
And I almost forgot, Virgina, North and South Carolina, and Georgia.
So you keep claiming. But even if so, that is one reference to slavery among twenty eight reasons listed. In the Declarations of the Causes of Secession submitted by four Southern states they mention slaves or slavery eighty three times, and hostility towards slavery is by far their most prominently listed grievance.