Because one can remain passionately neutral on the question of further enshrining slavery into the US Constitution.
I agree - he should have told the slavers to go piss up a rope.