Maybe because a phobia is an irrational fear, and therefore implies that the person who has the concern is in the wrong? Don't call me a homophobe, just call me a homoskeptical.
If a liberal finds me irrational I can live with that ;-)