I don’t see any insinuations or implications, either.
I see a metaphor. Sumner was beaten over a metaphor.
It's a metaphor until you are on the receiving end of it. Having slavery as your "mistress" is a very strong implication that you are being accused of diddling your slaves.