Go ahead. Ask women if they would rather see a man in a "nutter" or in a suit and tie. I'll go first:
Suit and tie.
Nice try, but that's not what you said.
It wasn't a "I'd rather" statement.
So, if you don't want to see men "that way" say so, but don't try to change the facts.