In that case, you are already breathing it in - so I don’t think a hand dryer adds much to the problem.
The force of your lungs is weak compared to the force of the blow dryer. I do not think you dislodge many bacteria from surfaces just by breathing.
Unless you can hold your breath for more than 5 minutes, you should enter with a face mask to filter the air you breathe in. Of course, your clothes will be covered in contaminants! Best bet, don't enter until long after the restroom is empty and the air has settled. Much to do about nothing, you take your chances in life every time you leave your home. My wife has a habit of immediately washing her clothes anytime she has sat anywhere outside the home, especially after using mass transit. She says the germs from the public will stick to your rear. Yuck.