Since slavery was illegal in the North, that would leave only the non secessionist slave states exempt in effect, to wit, Missouri, Kentucky, Maryland and Delaware. The only problem with that is that the "emancipation" proclamtion clearly states that slaves were freed in the areas of rebellion. I don't think that at the time the north invaded the south that the territories and states which you mentioned were in rebellion.
The four states were not, and thus slavery remained legal there until the 14th amendment. That was my point.