None
As there was no act that said it was illegal.
And where in the US Constitution was Congress given the power to criminalize slavery? Note that it took a Constitutional Ammendment to do that.
Bottom line: If the US Federal government can be held responsible for not outlawing slavery, then every person can be held liable for failing to come to the aid of another, whenever there was need. It is not possible to construct a logically-consistent ethical system based on any such premise.