"Am I in a position to make a difference?"
If the vast majority you have in Congress doesn't allow you to make a difference now there is no sense in staying.
We saw a similar rash of Democrat retirements and party switches in the ‘90’s when they saw that they were losing their precious power.