I think the criticism has to do with the way he expressed the frisk, not the frisk itself.
Frisking was way down the list, and was only done in a relatively small portion of “stop, question, and frisk” stops. By the time police got to the frisking point, the proportion of arrests was pretty high.
Heather McDonald had some very good articles which should give you a better explanation of what the policy actually was - or you could talk to some NYC non-criminals. Beginning with Giuliani, police were moved out into their neighborhoods and got to know the locals. It wasn’t random police just spotting people walking down the road and frisking them.