Nope! Superstar slugger Dave Winfield (my fave as a little kid) used to swing these ridiculously small bats (As a scrawny 13-yr-old, I used his model bat). 32 ounces. Rod Carew and Ozzie Smith swung even lighter bats, 29 ounces.
Babe Ruth used a 54-ounce bat.
Bat speed is very important. Those guys swinging a very light bat developed incredible speed.
Now, imagine if Babe Ruth had not pitched for the first 6 years of his career and had played everyday instead.