I have not scrubbed numbers, so please pound on me if I'm wrong, but I highly doubt that Bush's 3 mil-vote advantage was due to Gore voters changing sides. Just cannot see it happening. I think it's much more likely that those 4 mil Christians who supposedly sat home in 2000 actually went to vote this time around. (I'm strictly speaking in general terms here, realizing that this soup is made up of many ingredients.)
A quick glance at the exit poll numbers supports your view. In both 2000 and 2004, the same percentage of Democrats voted for Bush -- 11%. Of all voters who voted for Gore in 2000, 9% switched over to vote for Bush in 2004. But that's cancelled out by the 10% of Bush voters from 2000 who voted for Kerry this time. It doesn't look like there was any significant net erosion on either side.
You are probably right for the most part; but I was a Gore voter who switched sides. I can't be the only one who wised up after our country was attacked on 9/11.