Tories were the first. They were constantly and consistently anti-Founding Father. Abolitionists, while having many in their ranks who were anti-Founder, were not universal in their condemnation.
In contrast, for progressives the Founding Fathers are universally wrong and evil. Now, while the progressives were much closer to the tories in their universal hatred of the Founders, they were not tories. Tories were monarchists, not progressive ideologues. Just the same, progressives were not like the abolitionists either. Some abolitionists did start out hating the Constitution and/or the Founders. In contrast, Progressive elitism prevents them from ever changing their minds. Progressives believe they are the best of the best, and anybody who disagrees with them are stupid for ever considering alternatives. Progressives will not give up. Progressives will not relent. In the eyes of progressives, the Founding Fathers and the Constitution - limited government - is universally wrong and evil.
Progressivism didn’t exist until the 20th century, give or take 5 - 10 years or so right at the very end of the 1800s.
That’s why some of the best defenses of the Founding Fathers ever written or spoken were authored by the abolitionists themselves - who discovered that like them, the Founding Fathers many of whom were abolitionist also. Unlike progressives, abolitionists could and did have a change of heart because there’s no conflict between constitutionally limited government and abolitionism the way there is a significant conflict between constitutionally limited government and big-government statist progressivism.
For example, Frederick Douglass routinely defended the Founding Fathers themselves, and/or the U.S. constitution. Douglass’ defenses of the Constitution are some of the best ever written. Here is an example from 1860:
https://librivox.org/search?title=The+American+Constitution+and+the+Slave&author=Douglass
(Direct download)
https://www.archive.org/download/snf088_2112_librivox/snf088_amerconstitutionandtheslave_douglass_pa_128kb.mp3
Likewise, abolitionist George Livermore wrote “An Historical Research Respecting the Opinions of the Founders of the Republic, on Negroes as Slaves, as Citizens, and as Soldiers” - yes, defending the honor and the good name of the Founding Fathers.
https://freerepublic.com/focus/f-chat/4172958/posts
Progressivism is America’s cancer, not anything else. It begins in the Progressive Era.
The American Constitution and the Slave - public domain audiobook at LibriVox
Also, full original text: Mr. Douglass, in reply to Mr. Thompson
George Thompson was an abolitionist who thought he was a big deal until Douglass stood up and gave him the smack down.