Not surprising since women weren’t allowed to vote, Blacks were still slaves and only real property owners were allowed to vote in most places.....
The population of the US was 31 million in 1860, and 226 million in 1980. About 10% of blacks were free before the Civil War, but many of the free states did not allow black men to vote. I think the property qualification for voting had pretty much disappeared by 1860.