Opponents of slavery were gratified by the publication in 1840 of James Madison's notes from the Constitutional Convention, which they believed supported their antislavery constitutionalism.
Madison's notes really changed things for abolitionism more than we realize and that is how some abolitionists came to their conclusions. I've also been looking over Madison's notes and have seen the same things. These notes are also a part of the written record and cannot disappear. Douglass's speech doesn't float in the ether, he's referencing Madison's notes.
The end result are arguments on one side based on abstract theories of the hopes of these men, the Founders, versus the frailties of humanity across the 50 years of governing between the Constitution's adoption and when Madison's notes surfaced.
There are things of substance you pointed out that I don't agree with, but this is a looming forever argument since there's verifiable documentation on both sides that cannot disappear.
To me it’s an easy trilogy: the Declaration said “all men” without qualifying; the NW Ord, the very first national POLITICAL law, said no slavery in the territories; and the Constitution refused to identify slaves as property, only as “unfree PERSONS.” Pretty hard to maneuver around that, though of course states did. But the national direction of slavery was set from the time of the Declaration.
Once the argument was about the definition of a slave-—a person or property-—the South was finished.