Unless you look Arab or have a Muslim "shoe bomber" type aura, look at the bright side: the average ramp worker simply discerned immediately that you were of no threat. You passed his/her intuitive profile test, recognizing that you were a real American who didn't need to be shot on sight. :-)
Now if the ground crew looked suspicious (maybe I'd better not go there), then we have worse problems than a lack of gates/guards. What good will it do if you are prevented from accidentally accessing a plane, while those who continually work on or near aircraft, are Al Qaeda sympathizers. How can they screen people out, who merely have an interesting circle of personal jihad-loving friends (no paper trails or prior jobs at the local NOI chapter)?
Where was it (Logan?) that they reviewed employee files and discovered a bunch of illegals and/or criminals? What red flags weren't on paper, that they missed? Yeesh.