Given that the black population in Illinois quadrupled between 1860 and 1870 I guess he didn't have much to worry about.
A conversation was held between us after the negotiations had failed at Hampton Roads [3 Feb 1865], and in the course of the conversation he [Lincoln] said to me:--
But what shall we do with the negroes after they are free? I can hardly believe that the South and North can live in peace, unless we can get rid of the negroes. Certainly they cannot if we don't get rid of the negroes whom we have armed and disciplined and who have fought with us, to the amount, I believe, of some one hundred and fifty thousand men. I believe that it would be better to export them all to some fertile country with a good climate, which they could have to themselves.I said: I will go over this matter with all diligence and tell you my conclusions as soon as I can.You have been a stanch friend of the race from the time you first advised me to enlist them at New Orleans. You have had a good deal of experience in moving bodies of men by water,--your movement up the James was a magnificent one. Now, we shall have no use for our very large navy; what, then, are our difficulties in sending all the blacks away?
If these black soldiers of ours go back to the South I am afraid that they will be but little better off with their masters than they were before, and yet they will be free men. I fear a race war, and it will be at least a guerilla war because we have taught these men how to fight. All the arms of the South are now in the hands of their troops, and when we capture them we of course will take their arms. There are plenty of men in the North who will furnish the negroes with arms if there is any oppression of them by their late masters.
I wish you would carefully examine the question and give me your views upon it and go into the figures, as you did before in some degree, so as to show whether the negroes can be exported. I wish also you would give me any views that you have as to how to deal with the negro troops after the war. Some people think that we shall have trouble with our white troops after they are disbanded, but I don't anticipate anything of that sort, for all the intelligent men among them were good citizens or they would not have been good soldiers. But the question of the colored troops troubles me exceedingly. I wish you would do this as soon as you can, because I am to go down to City Point shortly and may meet negotiators for peace there, and I may want to talk this matter over with General Grant if he isn't too busy.
The second day after that, I called early in the morning and said: Mr. President, I have gone very carefully over my calculations as to the power of the country to export the negroes of the South, and I assure you that using all your naval vessels and all the merchant marine fit to cross the seas with safety, it will be impossible for you to transport them to the nearest place that can be found fit for them,--and that is the Island of San Domingo,--half as fast as negro children will be born here.
I am afraid you are right, General, was his answer; but have you thought what we shall do with the negro soldiers?
I said:
I have formulated a scheme which I will suggest to you, Mr. President. We have now enlisted one hundred and fifty thousand negro troops, more or less, infantry, cavalry, and artillery. They were enlisted for three years or for the war. We did not commence enlisting them in any numbers until the latter part of 1863 and in 1864. I assume that they have a year at least on an average to serve, and some of them two to three years. We have arms, equipment, clothing, and military material and everything necessary for three hundred thousand troops for five years. Until the war is declared ended by official proclamation, which cannot be done for some very considerable time, they can be ordered to serve wherever the commander-in-chief may direct.He reflected a while, having given the matter his serious attention, and then spoke up, using his favorite phrase: There is meat in that, General Butler; there is meat in that. But how will it affect our foreign relations? I want you to go and talk it over with Mr. Seward and get his objections, if he has any, and see how you can answer them. There is no special hurry about that, however. I will think it over, but nothing had better be said upon it which will get outside.Now I have had some experience in digging canals. The reason why my canal, which was well dug, did not succeed you know. My experience during the war has shown me that the army organization is one of the very best for digging. Indeed, many of the troops have spent a large portion of their time in digging in forts and intrenchments, and especially the negroes, for they were always put into the work when possible. The United States wants a ship canal across the Isthmus of Darien at some proper and convenient point. Now, I know of a concession made by the United States of Colombia of a strip thirty miles wide across the Isthmus for that purpose. I have the confidence of the negroes. If you will put me in command of them, I will take them down there and dig the canal. It will cost the United States nothing but their pay, the clothing that they wear will be otherwise eaten by the moths, the arms are of no worth, as we have so many of them in excess; the wagons and equipments will otherwise rust out. I should set one third of them to digging. I should set another third to building the proper buildings for shelter and the rest to planting the ground and raising food. They will hardly need supplies from the government beyond the first season, having vegetable supplies which they will raise and which will be best for their health. After we get ourselves established we will petition Congress under your recommendation to send down to us our wives and children. You need not send down anybody to guard us, because if fifty thousand well-equipped men cannot take care of ourselves against anybody who would attack us in that neighborhood, we are not fit to go there. We shall thus form a colony there which will protect the canal and the interests of the United States against the world, and at least we shall protect the country from the guerilla warfare of the negro troops until the danger from it is over.
Well, then, Mr. President, I said, I will take time to elaborate my proposition carefully in writing before I present it to Mr. Seward.
I bowed and retired, and that was the last interview I ever had with Abraham Lincoln.
Some days afterwards [6 April 1865] I called at Mr. Seward's office, reaching it, as near as I can remember, about two o'clock in the afternoon. He promptly and graciously received me, and I stated to him that I came to see him at the request of the President, to place before him a plan that I had given to the President for disposing of the negro troops.
Ah, he [Seward] said, General, I should be very glad to hear it. I know Mr. Lincoln's anxiety upon that question, for he has expressed it to me often, and I see no answer to his trouble.
Benjamnin F. Butler, Autobiography and Personal Reminiscences of Major-General Benj. F. Butler, Book Publishers, Boston, 1892, pp. 903-907