It is not as though asbestos was continually shedding into the air where people were. It was mostly behind walls anyway. So you are correct. Those who worked with asbestos. In factories and installing it. Yes, this was hazardous. Lead paint in homes and apartments was also innocuous, unless it was peeling off and kiddies were eating the chips. How often do you see interior paints peeling off? I have not.
I worked for about ten years in an MCI telecommunications facility that had asbestos in the floor tiles. The installers had to take measures against it. When drilling holes in the floor to install new telephone relay racks they had to put shaving cream from an aerosol can where the hole was drilled to trap the shavings, then scrape it up and put it in a plastic bag to sent it to a company that would properly dispose of it. In another facility where I worked there was blown in asbestos insulation on the poured concrete ceiling and anytime we needed to anchor the cable ladders and trays into the ceiling we had to get a professional company to assure the asbestos was properly contained.
Your account shows that proper precautions were taken regarding asbestos. But really, with the shaving cream. This could have been disposed of in the common trash. It looks like the asbestos was OK unless disturbed via drilling tiles and anchoring into asbestos coated beams.