Billbears, you completely misread what was stated, but I'll give you the benefit of the doubt as a fellow Southerner. ;-) Vermont abolished slavery in 1777, so at no time following statehood was slavery ever permitted. Where you misread the discussion regarding Congressman Collamer and his constitutent, is that they were talking about slavery EN MASSE down South, since none existed in Vermont. The doctor was "charitable" enough regarding some monetary provisions being made for slaves post-emancipation, but when Collamer mentioned that were they to be fully fair towards the soon-to-be former slaves and to take them in in direct proportion to their OVERALL population in the entire country at the time, it would swell the population of tiny Woodstock by 500 persons. The thought of 500 poor people of color coming to town put the good doctor off that notion of emancipation entirely.
I may have the statement backwards what I heard from a Black comedian said some 40 years ago, mentioning the difference of White racism of the North vs. the South. "Up North, they don't care how big you get, just as long as you don't get too close. Down South, they don't care how close you get, just as long as you don't get too big." Northerners could often afford to take very gracious and seemingly generous stances towards Blacks because they lived far away and rarely had to associate with them, but heaven forbid if they ever decided to move in and associate with them beyond a handful. Northerners were noted for sometimes acting in a far more visceral and reactionary way towards Blacks that were "in their face" than their Southern brethren.
Passed no segregation laws between 1865 and 1957.
1957: Barred public accommodations segregation
Source: The History of Jim Crow