On the other hand, my husband's mother is a New Jersey farm girl whose progenitors fought for the Union (at least the ones who had gotten here from Ireland already) and nobody around here holds it against him!
Actually, one of my ancestors was from New York City, and a bunch of his family stayed up north, so I probably have some cousins I don't know about who fought for the Union. I also had an Englishman ancestor who immigrated in the 1810s, to Newark NJ, with his father and brothers. A bunch of them stayed up north too. He moved to Alabama and did own slaves for a time, but sold them as he decided it was too much trouble and responsibility.
The historian Eugene Genovese said that slavery would have fallen of its own weight within a few more years, as mechanization and improved farming techniques made it completely obsolete.
Slavery was only economically viable as long as land was essentially free. A free man farming his own land is more productive than a slave who has to be watched, and only works hard enough to avoid punishment. He can therefore outbid a slave owner for farmland and still farm profitably