I'm not sure. I think his egomanical tangle with Al Franken over Franken's book really hurt him. I think the SBVT snub hurt him with his viewers.
Yep, Franken and the 'Who's looking out for you' book.