It may not be obvious outside the Beltway (and the DC columnists are certainly not helping in this regard) but if you live in the DC suburbs of Northern Virginia or Maryland, it's very common knowledge who in the neighborhood works for DoD, the Secret Service, DEA, foreign consulates, the FBI, NSA.... and the CIA.
You may not know exactly what they do - and they are extremely unlikely to tell you - but the fact of employment by this or that agency is generally not a big deal.
A showboating blowhard like Joe Wilson is exactly the sort of egocentric nitwit who'd tout his wife's employment to everyone he ever met, including cabbies, waiters, doormen, trash collectors, and of course reporters.
It's predictable that the DUmmies and their reporter/columnist allies would use this general naivete about "DC as a company town" as a propaganda lever. Wilson and his fans are counting on the naivete of the American public.