It’s right to be nervous.
Incumbent presidents usually win reelection because voters basically give them the benefit of the doubt to try again.
That said, I think the level of anger will really help Romney this year. At the same time, he needs to present an alternative. From what it sounds like, he is preparing to do just that with an ad blitz in a few weeks as well as coming out strong with his economic plans.
Post-Nixon, Ford, Carter and Bush lost reelection while Reagan, Clinton and Bush Jr. won reelection. Ford might not be fair to count. But, still, in the modern era, there seems to be about a 50/50 chance of an incumbent winning reelection. I'm really not sure comparing much further back than that is valid. The presidency and how the public views it has changed SO MUCH since the Nixon era.
What seems more true to me is that the most charismatic candidate wins presidential elections every time. That remains why I fear for Romney's chances against Obama, the difference in how they poll on likability.
The debates are where people really make that judgment. Which is why I think discussing polls or anything else before the debates is pretty much meaningless pontificating. If the debates are a draw, then everything up until then might matter more. But if there's a clear winner in the debates, it'll basically cancel out everything that's happened up until then. I think a candidate under 50% can go above 50% easily almost overnight if they win the debate.