They weren't...yet. A condition that the Confederacy was anxious to take care of.
And the reality of the time is we did nothing to stop slavery in the Caribbean and Brazil. Apparently our moral high horse only extended to those people who could be financial threats to the crony capitalist power structure still running the Capitol today.