The most common excuse I hear in response to this is “The parties were different back then.”
Many Neo-Confederate freepers also believe that. They would have us believe the parties magically switched ideologies at some point after their beloved Southern rebellion. Modern Democrats who refuse to associate themselves with slavery and Jim Crow believe the same. It's a pant load of bull.
Democrats also say that the Southern democrats were all conservative. Some of them were, because almost everyone in the South was a democrat at the time. Many were socialists, all were bound together by their racism which is the reason it took so long for the conservative ones to leave the party that was socialist on the national level, see that couldn't go joining with those Negro-loving Republicans.
Issues have evolved, the 19th century was a very different time, but IMO Federalists-Whigs-Republicans were always the party of commerce and the Jeffersonians-Jacksonians-Democrats were always the party of populist BS and prior to the civil war, slave power (The Whigs were divided on that which is why they collapsed). They are still the party of racism. They have been the party of socialism since 1896 when the Bryan Marxist Labor faction (there were also a few Marxist Labor Republicans but they never controlled the party) won control from the Grover Cleveland Bourbon Democrats. Cleveland is one of the only Dem Presidents that wasn't a total piece of worthless crap.