Is there a way to stop that?
Short of not having an e-mail address? Not really. If you use your e-mail address at all, SOMEone will have it in their address book, guaranteed. And that means a virus on their system can send in your name.
In an ideal world, everyone you communicate with via e-mail would have the latest antivirus software running on their system, but that's not the case (unfortunately).