I have quizzed land surveyors about this. There is a minimum altitude (FAA rule) that planes and helicopters can fly over your property. Anything less than that could be considered trespassing. I don’t remember the altitude, but its certainly higher than shotgun range.
This guy is going to get in trouble because of the discharge of a firearm in the city limits...but its not ‘ok’ to hover around over other people’s property. Especially as drones get bigger and have more powerful blades - if they fall or fly at you, it could actually be dangerous.
1000 feet is the minimum over "populated areas", 500 over rural areas.
I am guessing that roaming frequency signal jammers are going to become available in the near future to address snooping. But like you said, I sure hope nobody gets hurt from falling drones.