Paul must have missed your memo when he wrote to Philemon.
Here we go. The Bible is on the side of slavery. You are the saint, I am the sinner.
Paul wrote of the slavery of his time.
Slavery in the time of the insurrection was legally different, in some ways harsher, in some ways less so. No Jubilee for the southern racial slaves.
Legal differences would make blanket application of Paul’s advice on one insititution to another at least questionable, and at worse unfair.