When the Constitution was written, slavery existed not only in the South but in the Middle Atlantic states, except Pennsylvania, where it had only recently been abolished, and in the New England states, except Massachusetts. In that period, there was not only chattel slavery of African-Americans, but also indentured servitude of whites of British and Irish origins. That the Founders were not proud of the institution is evidenced by the banning of slavery in the Northwest Territory and the cessation of the African slave trade after 1808. Their failure to eliminate chattel slavery at the founding of the Republic was a serious error that led to the Civil War 72 years after George Washington took the oath of office as the first President.
Slavery should have been abolished in 1788 by mutual consent, not in 1865 after the deaths of hundreds of thousands of Americans and the ruination of much of the South.
You forget that the South insisted that it remain, the efforts of some to end it notwitstanding. George Mason was one.
As those 80 years went by, the South pretty much controlled that agenda and the courts, even got the Fugitive Slave Act passed.