It was reported in the papers of the time. You have no proof that the quote is not accurate.
And neither do you.
Oh, well since we are doing that:
"If I thought this war was to abolish slavery, I would resign my commission and offer my sword to the other side."
And here's another:
"After due consideration the administration determined upon the arrest of the Chief Justice. A warrant or order was issued for his arrest. Then arose the question of service. Who should make the arrest and where should the imprisonment be? This was done by the President with instructions to use his own discretion about making the arrest unless he should receive further orders from him."
Right back atcha.