Here's a map of population density in 1860.
The free population of the free states was 18.5 million.
There were 5.5 million free people in the Confederacy and another 2.5 million free people in the border states.
Obviously, the free states imported more than the slave states and paid more in taxes.