Yes, I pulled carpet out of all the bedrooms in the old house and only kept it in the living room/hall. I still like walking on a carpeted floor. But you're right, carpets do hold dirt like there's no tomorrow.
The new house has carpet in all bedrooms and the living room. If I want hardwood, I'm going to have to remove the carpet and install the hardwood since there's only underlayment below the carpet. Perhaps in a few years.
Honestly, I don't think carpet is that big of a concern with mold, though dust and dust mites may be an issue. Mold is more likely to show up in an area that gets wet and stays wet.
BTW, you may have stumbled onto something else that can be associated with the increased occurance of asthma - carpet.