I would be careful about washing cloth masks. Certainly they should be washed before wearing the first time, but washing too much loosens the fibers, which can be breathed into the lungs.
Best to hang them places where they get sun or use one of those ultraviolet things used to clean cell phones.
Perhaps you are right about the loose fibers...but I make a point to not breathe through the mask. I breathe exclusively through the nose, which I leave fully exposed at the top of the mask.