First, UAVs can be knocked down fairly easily with a concerted effort--and the right weapons. As I recall, NATO lost 16 UAVs in Kosovo, including U.S. Predators and a high-speed Franco-German model (CL-228?). Most of the losses occured in an area where UAVs operated frequently. The weapons of choice were anti-aircraft guns (23/60 mm, along with heavy machineguns), or man-portable SAMs. One area in southern Kosovo was nicknamed "the triangle of death" because so many losses occured in that area.
Secondly, the Serbs understood the importance of UAVs before the war began, and made a concerted effort to engage them. They even staged an air defense exercise to refine their anti-UAV tactics. However, despite these losses, the UAVs still "got through" and accomplished their mission.
I'm not sure how much credence I put in these Iraqi UAV reports, but clearly some bigwigs are worried. My biggest concern is that our air traffic control/air defense system is simply not geared for slow-moving, low-flying targets. A UAV with a Predator signature won't even register on radar at low altitude, because of its low RCS, its slow speed, and ground clutter. Additionally, we have virtually nothing to engage them with, aside from a few Stingers and Avengers now deployed in the D.C. area.