In fact, as you are no doubt aware, the "rule of four" is itself not something that is actually set forth anywhere in the Supreme Court's rules or otherwise formally "codified." It is merely longstanding Supreme Court practice. So to say that, with recusals, some sort of "rule of three" would apply is a bit of a non sequitur.
When has such habit been broken?