It’s not talc itself (it’s one of the softest known minerals). It’s the associated asbestos fibers that are hazardous.
As someone who had asbestos exposure in a shipyard during the 70s, and has been informed on the matter, you are correct. And I’m fine.
BTW, they love throwing the word “toxic” in there. Asbestos is just about the most inert thing you can find. It’s literally fireproof.
What does happen is that it’s an irritant. Because of the fibrous structure it gets stuck places it shouldn’t be and the body can’t expel it. With regards the lungs, enough exposure results in asbestosis which may then progress to mesothelioma.