I'd also remind you that seven years after the so-called "Fudge" remarks, slavery was all but ended and Lincoln called for voting rights for blacks. It was the last public speech he ever made. In the audience when he made that call was a man named John Wilks Booth who vowed then and there to kill Lincoln over the issue of "Negro" equality.
Like I said, in that day it could be very dangerous to say what you really think.
Kind of reminds me of some places in our country today. H.K. Edgerton has been seriously, physically threatened when he set up a booth at a Kwanzai festival with pictures of his Confederate ancestors. My brother was there (as part of security at the festival), and told me a crowd of brainwashed Blacks were going to kill one of their own for telling people the truth about his lineage.