No, I believe many of the Cardinals know that they, too, should join the brave four. Which ones are being prudent in keeping quiet (for now) and which ones are being cowards, only God knows.
No cardinal is prudent if he remains silent with a heretic pope.