There were PLENTY of slaves up north - and I don’t mean freed slaves.
Yes, it 'twas a problem . . . but turning so many of them to freedom caused all kinds of logistical problems, especially since most of them were illiterate and had few skills.
Even for those Negroes who fought for the North, equal pay didn't come for many years after the war was over.
Lincoln was on to something with the Liberia idea, but it wasn't handled too well. Anyway, sending them back to a continent that sold them wholesale into slavery wasn't too appealing to the Negroes.