Rumsfeld wasn't a deciding factor. It was those people who call themselves conservative who decided teaching the GOP a lesson was more important then our national defense. It was the people who call themselves conservative yet claim there is no difference between Democrats and Republicans. It was those people who put their single issue ahead of all else.
You might have an argument had Rummy stayed on. But he left anyways, so why lose on both accounts?