The point is that a baseball player fields, throws, and hits. (With the pitchers getting an excception). If a guy is a great fielder, and nothing else, he won't get in. If a guy is a great hitter and a poor fielder, he can get in. But, if a guy is so bad in the field that he can't even play in the field, then that ought to count for something.
Do you take the same limiting view with respect to relief pitchers making the Hall of Fame? Like the DH, the closer is a specialized role that has developed in baseball's modern era. If it's an integral part of the game that these positions exist today, then shouldn't the best of them be acknowledged?