Southie, silica can just blow in if there's lots of finely ground loose silica around. The fact that silica was found can be used to suggest something much more useful like where the envelopes were filled.
True, if silica was the only mitigating factor. However, the anthrax attacks were revealed to aerosolize readily, which indicates professional mixxing, preparation, etc., rather than just random contamination (which would give you some significant percentage of anthrax spore clumps).
It would also be rare for contamination to be only one compound (e.g. silica). You'd expect contamination to celebrate a bit more diversity (e.g. pollen, human skin, dirt, and other common non-clean-room substances that are ubiquitous).
So there are *multiple* factors that evidence professional weaponization of the anthrax in a clean room, followed by reasonably clean packaging of the anthrax into the letters.