Overuse of antibacterial soap can kill good bacteria that live on your hands, potentially leading to skin warts. Temporary and not a big deal, but certainly unpleasant.
However, there is no mechanism by which your immune system “gets weaker” based on a lack of exposure to the same number of pathogens. There just isn’t a way for what you describe to happen.
I’m no expert, but I have a different opinion. Immune systems need practice, although serious pathogens are not recommended.
Natural viruses are quite like computer viruses.
Your body can get out of sync with natural virus populations.