Really? I have only heard men use that in terms of women whom they think are ugly.
I've used that term for a number of men.