Atheists have no beef. If atheists truly do not believe in God, then they should be able to walk away from this debate. The fact that they need to justify themselves is a strong sign that they have strong doubts about their beliefs.
More to your point, Bush did nothing to espouse Christianity in office. In fact Clinton was the president who was constantly around Christianity -- photo ops of him going to church, pictures of him carrying a bible and so forth. If anything, it was the Clinton Administration that should have made the atheists irate, not Bush.
That was certainly the case for myself, but I suspect I'm in the minority as regards atheists as a whole...they tend to be quite left-leaning, with the notable exception of those who have Objectivist & Libertarian tendencies.
Except that even atheists can recognize genuine faith. That's what irritates them. Not religiosity.