I feel much the same way. It is his behavior after he left the USN that I take umbrage with.
I leave it up to those who served with him both before and after his capture to pass judgement on him for his behavior. And there are a number who have.
As I said, I only served under him as a trainee for a short time, and honestly, he never said much of anything that I could hear. I did see him exhibit some odd behavoir once, as he was just standing at a scuttlebut in hallway of the squadron admin offices, and he was just completely motionless staring at the water coming out for a long time...I thought he might have been waiting for it to get cold, but it was a lot longer than that. He looked like he was simply zoned out on the water stream.
In the years since, I have wondered if that was a holdover from his days of captivity, since that was only a two years after he was released, but it was odd to see.