You must have missed Nancy Pelosi being speaker of the house.
She was not speaker for the 108, 109 and 112th Congress. She was speaker for the 110 and 111th Congress. So for 6 of the past 10 congresses she was NOT the speaker and Dems were not the majority.