It does not seem to me in the slightest implausible that a gene might cause people who carry it (whether male or female) to be unusually-strongly attracted to females; while this would likely reduce reproductive success rates for females with the gene, it might increase it for males sufficiently to offset the reduction in females.
It is also not implausible that another gene might cause attraction toward males, again with similar (but sex-reversed) consequences.
Since women can only be gestating one offspring at a time, they are a much more valuable part of the "natural selection" process. Any trait that caused a percentage of females to be less likely to reproduce would be selected against.