Any team who loses a star kicker is going to be in trouble, field goal kickers put up more game winning points than any other position in the game. But other than Viniteri I think the Pats can survive any injury to the team, Brady would probably be the second most painful loss because he really is the perfect QB for Belichick's system, but I'm sure the One True Bill could build a new system around a different QB if he had to.
Field goal kickers are like closers in baseball. The good teams have the best of them.