You're more right than you may realize. Asbestos occurs naturally in most soils of the mountainous regions of the west. If this is allowed to establish a precedent, we may have to vacuum clean California daily!
The reality is that this whole thing is likely an attempt to get funding reauthorized for super-fund.
It isn't all that common, it is pretty much restricted to the sepentine belts in California, both in parts of the Coast Range and the foothill Sierra, and it also is found in very "basic" rocks in parts of Canada, and even in Montana. But a lot of this stuff outside the serpentine belts is the "long-fiber" variety. None of it is very good for humanoids, (or doggieoids), but I suspect the entire danger is way over exagerated...as usual.