Edgar Martinez was actually a pretty good defensive third basemen before his injury proneness forced him out of the field. I don't think that necessarily says anything about how much of a baseball player one is. Does it somehow make you a better ballplayer if you do play in the field, but are a below average fielder? Does that help the team more than a good DH?
As a Red Sox fan who watches Manny Ramirez all year, I can testify to that. He's an amazing hitter, but a running joke in the outfield. A decent arm, but no sense of what to do out there. He once dove to cut off a throw from Johnny Damon that after for no apparant reason, he dropped a ball that would've been the final out of the game, allowing the tying run to score, he had the infamous tumble in the World Series. I could go on forever.
The point is that a baseball player fields, throws, and hits. (With the pitchers getting an excception). If a guy is a great fielder, and nothing else, he won't get in. If a guy is a great hitter and a poor fielder, he can get in. But, if a guy is so bad in the field that he can't even play in the field, then that ought to count for something.