The small surface to air (SAM) scenario is one that has our gov't terror analysts sweating bullets IMHO. There are hundreds if not thousands of SA-7, 13, Singers and Redeye shoulder launched SAMs unaccounted for. Terrorists have had limited success in Iraq with them as well as some have been shot a commercial aircraft in Africa (where I don't remember). And you are correct, the number of places that are accessable to use these little buggers is almost unlimited.
How many airports do you know of where people can pull up to fences or in dirt lots adjacent to airports or cargo areas and wait and watch for planes to take off and land? There are virtually no "no parking or stopping" signs, barriers or watchful eyes where they should be.