There were two people present. One died a week later. The other wrote up his recollections of the meeting in detail. No credible reason whatsoever exists as to why those details would be wrong. Live with it.
It's inconsistent with what President Lincoln said before. There is no reason to give it credence.
"Congress may apprpriate money and otherwise provide, for colonizing free colored persons, with their own consent, at any place or places without the United States."
-- A. Lincoln, 12/01/62
You can't show in the record that Lincoln's position ever went beyond that.
What the record shows, in fact, is that he dropped this position entirely and supported full rights for blacks.
Walt