I substituted ‘Immigration’ for ‘Slavery.’ ;)
Like the Democratic party before the Civil War, the Whigs were a “bisectional” party that drew voters from both the North and South, explains Philip Wallach, a resident scholar at the American Enterprise Institute.
“Both parties therefore had an interest in keeping IMMIGRATION off the national agenda as much as possible,” says Wallach. “But in the case of the Whig party, it just couldn’t find any way of dealing with the IMMIGRATION issue that would satisfy both its Northern and Southern wings.”
https://www.history.com/news/whig-party-collapse
Which is why the American Party, also known as the Know Nothing Party, became a serious player as the 1850s progressed. The decade was marked by new parties that appeared as the Whigs and Democrats both disintegrated.