Yes, this is an oversimplification on the order of "See Spot run!"
There is no one "favorable" or "detrimental" in most individuals. There are thousands of benign (here and now), slightly detrimental (here and now), slightly beneficial (here and now), etc. The range is huge. That seems to be the point you are missing.
Try a close look at sickle-cell anemia. One single trait. Bad news, right? Right, except that it provides some resistance to malaria. So, the efficacy of this particular trait (out of millions) depends on--is there malaria here? If there is malaria here, then you have a slight advantage in that one area, while still maintaining the disadvantage of the anemia. If there is no malaria, the slight advantage of malaria resistance does no good, and the anemia is still detrimental.
Multiply this by thousands of traits. The folks with the best overall adaptations for here and now (and here and now is always changing) survive and reproduce a little better than those who do not. Toss in a few million years and stir well.
But forget the mathematical models until you have a handle on the variables. If you can't figure out all the variables, and correctly model them, your mathematical models don't mean much in the real world.