I use the age and racial data provided by NEP, which is used by all major news networks, AP, and the NY Times.
You may be using some parts of the Census Bureau election data, which can be substantially different.
According to my math:
2012 - Obama’s white vote went down 5.5 million.
2012 - Romney's white vote went up 1.4 million compared to McCain.
2012 - Total white voters went down 4.2 million.
Many folks at Free Republic insist that 4 million Conservatives stayed home in 2012.
I don't think the data supports that conclusion.
I think it is much more likely that 4 million white Obama supporters stayed home in 2012.
A definitive answer to this question can actually be found in the raw NEP data.
But that's behind a pay wall.
Romney got fewer than a million more votes than McCain, although he was running against Jimmy Carter’s second term, in an election that republicans couldn’t lose, and on the verge of a depression.
Mitt did that while easily winning Independents yet losing the election, which is almost unheard of, Romney depressed the republican vote is what it seems.
Although Romney got 59% of the white vote, that was close to the 58% in 2004, and much less than numbers like 67% and 64% and white turn out was low.
If his white numbers went up greatly as you say, and he did great with non-republican (the independents), then someone stayed home among the base.