You're correct, but in my defense, I've never worried about being "manly" either as a boy or as gentleman. My parents raised me as a gentleman. In defense, I don't remember ever using the word "manly."
It was Carter who used it.
And he was mocking Reagan voters—i.e., asserting that they voted for Reagan because they perceived him as more of a cowboy.
A more appropriate word would be “masculine.”
Or perhaps, “non-repulsive,” “non-treasonous,” “non-idiotic,” “non-lying-crypto-Marxist,” or “non-anti-Jewish,” or “non-pro-abortion.”