Probably this would have been regarded as an extraconstitutional affront to private property rights. Nowadays we don't normally think of people as "property" (Unless of course we are referring to an unborn child) but it was apparently a common place view of that time period.
BTW, Lincoln agreed fugitive slaves must be returned. He just insisted on reasonable due process to determine they actually were slaves, which seems reasonable.
Had they given him a chance, they may very well have discovered that Lincoln would have been to them far more reasonable than his rhetoric implied. The problem was they were too sensitive and his oratory was too moving.
He was a D@mned fine thinker and writer and speaker.
It was his election as such they objected to, not anything he did himself.