It's very possible to know whether every ATF field agent was absent or present that day, if you go down the roster and account for each individual. It's time consuming and possibly difficult, but not impossible.
As to whether it's "clearly a case of some guy saying that and an irresponsible (or lazy) reporter, etc., etc.," I have no idea why you'd draw that conclusion. Charles Key's commission found that ATF agents were absent from the building. Whether that means some, most, or all is unclear from the sources I've cited, but my memory of the reports at the time is quite clear.
Now, it may be that both Mr. Key and myself are paranoid kooks. If we're going to be rigorous, it's best not to disregard that possibility. =]
I'd like to see the Key commission's exact wording. It would make a world of difference if "some," "most," or "all" were missing.