At a practical level how do you deport 4 million people without their complete support? Face it, even if Lincoln and Davis both were hell bound to kick every black face off the continent neither one could possibly have accomplished it. The logistics involved with such a move would be staggering.
I don't know. The buffalo didn't want to go, either, but Sherman and Sheridan somehow managed that.
It would certainly have been a giant "Trail of Tears" -- but there was also the precedent.