True. I was just thinking about how Romans used to consider anyone who wore pants (rather than a civilized kilt or long tunic) to be barbarians.
But, crossdressers are drawn to the clothing that is culturally associated with the opposite sex. It's not really the clothing that attracts them, per se, but the opportunity to emulate the opposite sex.
Of course, these days there are only a few articles of clothing in our society that are considered exclusive to one sex.