The anecdote from the Congressional Globe doesn't say that at all. It's about a Vermont farmer being willing to compensate southern slaveowners for their slaves, but not wanting them in his town.
Why don't you tell us then what Representative Collmer said then?
Mr. [Jacob] COLLAMER [R-Vt.]. As the gentleman has called me out, I may be allowed to say that the inhabitants of the town were about three thousand, and the proportion was about one to six.
COLLAMER, Jacob, a Representative and a Senator from Vermont; born in Troy, N.Y., January 8, 1791; moved with his father to Burlington, Vt.; attended the common schools, and graduated from the University of Vermont at Burlington in 1810; served in the War of 1812; studied law; admitted to the bar in 1813 and practiced in Woodstock, Vt., from 1813 to 1833; member, State house of representatives 1821, 1822, 1827, 1828; State’s attorney for Windsor County 1822-1824; judge of the superior court 1833-1842; elected as a Whig to the Twenty-eighth, Twenty-ninth, and Thirtieth Congresses (March 4, 1843-March 3, 1849); chairman, Committee on Manufactures (Twenty-eighth Congress), Committee on Public Lands (Thirtieth Congress); appointed Postmaster General by President Zachary Taylor 1849-1850; again judge of the superior court of Vermont 1850-1854; elected in 1855 as a Republican to the United States Senate; reelected in 1861 and served from March 4, 1855, until his death in Woodstock, Windsor County, Vt., November 9, 1865; chairman, Committee on Engrossed Bills (Thirty-fourth Congress), Committee on Post Office and Post Roads (Thirty-seventh through Thirty-ninth Congresses), Committee on Library (Thirty-eighth and Thirty-ninth Congresses); interment in River Street Cemetery.
The Representative from the State of Vermont confirmed the anecdote from the Representative from Wisconsin. That's exactly what he says