After Fort Sumter, the controversy pretty much ended. The opposition in the North afterwards was confined to dissent over the conduct of the war and in 1863 some pretty hard core riots to protest the first military draft in this nation... (good thing to protest in my book)
Your observation about not wanting “slavery” to spread is spot on and evidenced by the awful racial incidents between 1865-1930 in Northern or Border States, the rise of the KKK in the midwest (as a response to blacks moving into northern states to get factory jobs), and the ‘soft’ discrimination in places like Michigan and California.
As you point out, the North and the West saw widespread discrimination. The 1863 New York draft riots degenerated into an anti-black pogrom. There were lynchings of blacks in such un-Southern places as Duluth, Minnesota and Wilmington, Delaware. Segregation of schools by race was not uncommon, and required by law in Indiana and counties in New Jersey. Racial covenants in housing were common outside the South, and effectively confined blacks, and sometimes Asians, to designated neighborhoods.
The rise of the Ku Klux Klan in the North and West was more motivated by anti-Catholicism and anti-Semitism than anti-black sentiments. States like Maine, Colorado, and Oregon, where the Klan was strong in the 1920s, had few blacks. Oregon even attempted to close private schools, most of which were Catholic, in that decade, but the courts overruled the state's legislation.
The people who write history books would have us believe that the widespread Northern opposition to slavery is based on moral objections to the practice. The truth is that most Northerners did not give a crap about the suffering of black people. They opposed slavery mainly for two reasons. The first is they absolutely hated the idea that someone could get free labor when they had to sell their own labor to make money. In the regions that became the heavily labor Unionized North, slaves were seen as "scabs" who would take the food out of the mouths of white people who had to work for wages to earn a living.
The other primary reason for Northerners hating slavery was a general hatred of black people and a desire to keep them far away from white people. The Laws of Illinois and other states illustrate just how much Northern white people hated the black people, and how they had no concerns about black people being forced to work, but a great deal of concern about black people being near them or their families. In Illinois it was legal to grab undocumented black people and sell them into slavery down South.
The actual truth about Northern opposition to slavery is very ugly, and people always try to cover up the real truth by pointing at the collection of Abolitionists of that era, but the abolitionists represented a teeny tiny component of the population at that time. Most people hated slavery because of labor competition and a general hatred of having black people around them.
Also, slavery wasn't going to spread into the territories to any significant degree. Cotton could not be grown in the territories except in New Mexico, Arizona, Nevada and California, but only if irrigation systems which didn't exist in 1860 were used.
Cotton could not have been grown in the territories until the next century. There was no real threat of slavery expanding into the territories. It was economically unfeasible.