Unless you're in a place where people forfeit their 4th Amendment rights (airport, courthouse, driving in a car down the street, etc), you have to have reasonable suspicion for the stop itself. The Fourth Amendment prohibits searches AND seizures. Call it a frisk or a search--where is your reasonable suspicion to stop (seize) me?
There are many factors that can make up reasonable suspicion. To name a few: furtive hand movements, evidence that you are ‘casing’ a location, change of direction upon seeing police, being in a drug-prone location, matching an eyewitness description for a crime recently committed, being pointed out by a witness. There are a lot of factors that when added up in the totality of circumstance could very easily warrant a stop and question. And, if advisable, a quick frisk of outer garments for weapons.