You could say the same for many car drivers - blissfully ignorant of other operators, run stop signs, fail to indicate etc. why don’t they illicit the same abuse?
They often do ... it's called "Road Rage".
As a rider and a driver I can safely say it is that segment of riders who don't ride with consideration and/or follow traffic laws that make it so hard for the other ~15% of us.
More years and pounds ago than I care to admit, I commuted to work on my road bike. It was a hilly 17 miles each way and I equipped myself with proper lighting to be seen and a mirror to see. One ~4 mile stretch of road was particularly dangerous. It had two narrow lanes with no more that 6" of pavement between the white line and a drop off into a ditch.
Cars commuted on it too and I always did my best to not hold them up. I learned to "hold my line" there like you cannot imagine; I would ride that white line and wave cars past me. Did it for years and the ONLY close call I had was a trailer with a wheel base wider than the pick-up pulling it - a somewhat understandable error - his trailer fender passed me about one foot away. All of the drivers were considerate, careful and respectful. So was I, and I think they knew it; they got used to me being there and we looked out for each other. As it should be, but often isn't.
I think bad drivers illicit plenty of abuse (generally in the form of complaints at the water cooler). The difference I see is one, there is more of them so we tune them out and two, their actions are not nearly as likely to get them killed and another person jailed/sued when an accident occurs.