I don't know where you got your information, but you are absolutely 100% wrong on that. If a hunter accidentally shoots someone minding their own business in their yard, they are civilly liable, and in all likelihood criminally liable.
They may be liable, but it happens,and plenty of times the Hunter is unaware, and there are times that the authorities do not know who did it. There is no solution that will please everyone. Years ago people did not live that close to where the Hunting areas were,now you have developments that border woods that Hunters are allowed to hunt in. Sometimes people have their pets accidentally hit by over zealous hunters. Hunters cannot guarantee that their shot that misses the Deer will not hit a nearby residence. Are they supposed to be blamed for this and called careless even if they follow the rules? Did you ever see vehicles that park near a driving range even one equipped with a net, they are hit and damaged by errant golf balls. The driving range does not usually pay much, your car insurance does.