I'm in that group with you. Those who point to the large number of stars and planets haven't combined that with the othetr side of the calculus, the probability math on life arising from inanimate chemistries.
Frank Drake made a go at actually calculating the odds, but it is really just an informed guess. And all knowledge in biology since Drake's equation would make it less likely, not more. But most people just formulate it as a belief, as in, "there just has to be something out there."