Sandy Koufax is considered one of the best despite a short career and less than impressive career stats in terms of wins. The same holds true for Gayle Sayers who spent only 6 years in the NFL. Brady is one of the best regardless of what happens.
As it happens, I was a big Koufax fan in the middle 1960s. But Koufax while a baseball hall of famer would not be listed among the top two or three pitchers in MLB history for that very reason, too short a career. As I said, Brady is a hall of fame QB today. He is not among the top three in NFL history.