I don't know of any other Vietnam War prisoner, who was accused of aiding the enemy. Granted this has not been proved, and some of the prisoners support him, but it fits his persona with all his other immoral actions, starting with the way he treated his wife.
Reagan supposedly couldn't stand him. It was Nancy he cozied up to.