That's pure argumentum ad hominem. You privilege adherence to the Union cause as the only litmus test of honor. That is the blackest sort of bigotry on your part, and it's intellectual dishonesty as well, inasmuch as you deploy it like a fig leaf, to cover the nakedness of your refusal to discuss the issue.
General Taylor was a contemporary and an eyewitness to the views of Thaddeus Stevens. That Taylor was civil enough to take an interview with someone like Stevens without shooting him doesn't matter to you. All that matters is, which side he was on in the War between the States.
Your bigotry didn't really need my highlighting, it's a veritable lighthouse by itself; but I'm happy to supply a little neon if it helps illuminate things better.