Exactly how many blacks are there in Vermont even to this day?
I'd say a pretty good reason for the low number is the fact that Vermont had no slave population to start with.
Of course, it’s easier to elect Charles Sumner when you don’t have to live with the consequences of his policies. I’m sure Detroit was filled with people in, say, 1850 who were horrified by slavery in the South and who were, relatively speaking, liberal on racial issues. Of course, Detroit then was a virtually all white town. When it ceased to be all white, the result was race riots. And eventually, the whites packed their bags and moved the hell out of there.
More recently, the nearly all white town of Lewiston, ME had an influx of Bantu tribesmen, and a town which had never had racial strife before suddenly had racial strife.
Blacks made no significant effort to move to these supposedly racially tolerant places such as Vermont after Reconstruction because the white folks there weren’t any more tolerant at all once blacks actually started showing up. They did move to some large northern cities when industrial jobs became plentiful. They fought it out with the whites, and when the whites found out that the political powers-that-be, and the media and the courts, were going to side with the blacks, they simply moved to another part of town or left the city entirely.