Are you suggesting that society should regard it as being perfectly okay when a cross-dresser misrepresents themselves to you in such an expert fashion that an intelligent man (or woman) can't tell the difference? Some of them are quite skilled in modifying their appearance, and most people would be quite hard-pressed to tell for sure.
Should such an expert liar have absolutely zero responsibility in the matter when they are dealing with such a potentially volatile issue as interpersonal sexual relationships?
It doesn't take a locksmith to tell the difference between a combination lock and a keyhole.