1. One of the ways you can get the coronavirus is by touching a surface and touching your face, so wearing a mask improperly can increase your risk of getting disease.
You do not get a virus by touching anything, unless that is an entry point to your body. Thus it must be assumed people with masks are putting their hands in their mouth or up their nose.
In addition, rejecting masks since some improperly wear them would be like rejecting guns since some also misuse them.
2. It can also give you a false sense of security.
Likewise seat belts and helmets, so to be consistent with rejecting something that is overall beneficial due to what can or sometimes do happen negatively, then all such that do so should be banned.
3. The data does not show it helps.
Or that it does not, and common sense tells you it would help prevent transmission as well as reception, or at least the amount of it. See