They keep their clearance and access because they often made (or were involved in making) major decisions on highly classified issues of long duration. Some have been in and out of government under multiple administrations; each time returning to a more important position and usually continuing to work the same issues. In short, they are a repository for understanding the long term and short term reasoning behind decisions. This is something that is not often written down explicitly. Biographers and historians, among others, often spend a lot of time trying to reconstruct this because of the insights they give to the personalities involved in the decisions and the factors those personalities considered important.
The deal with them losing their clearance and access is not just a loss of access and a blow to their personal prestige and marketability. The experienced advice they can provide is lost and, if properly observed, none of their friends and colleagues who still have clearances and access can talk to them about ANYTHING classified, historical, recent, related, unrelated, tangential, etc. Nothing.
Suddenly, like Will Rodgers, all they know is what they read in the newspapers.
If no one with clearance can now share their intelligence with him we should watch him closely to see if he slips up and reveals something he’s not supposed to know.
That's technically true even if they have their clearance unless they have a need to know.