by the standards of today, 99% of the American population in 1860 would have been considered virulent racists.
That may even have been true, just a century ago. IIRC, the Klan reached something of a membership peak in the 1920s or there abouts, and membership was definitely not limited to residents of Southern States...