Right, clearance or not you can have ALL access terminated.
The get to the information, you must have all three:
1. Clearance
2. You must be listed on the access list for any given repository of classified info.
3. Even if you have clearance and you are on the access list, you still get nothing unless a need-to-know can be documented.
There are often multiple "access lists" too.
When I did the dastardly deep of getting married to a foreign national, the SSO pulled my access to SCI. My TS Clearance remained intact as did my Crypto Access.
I could have transferred directly to a Signal Corp unit and continued to work my MOS as they only needed the Clearance and Crypto access in their facilities.
Within a SCIF even having the needed access doesn't grant access to everything as you said. You must have the "need to know" for that material. Curiosity isn't "need to know."
Within each SCI there could be several code word projects, and even sub categories within those. Each project and sub had it's own "Need To Know."
Another example was the comm Center access list. Every one of our unit in operational MOS's had TS Clearance, Crypto Access, and were read on to SCI's. They still had no access to the comm center unless they were on that separate list.