Yeah, but Montoya had spent his entire life preparing himself for that fight. And the use of any weapons change the equation away from size and toward skill. With firearms or other missile weapons, when the two combatants are equally skilled the larger one has a slight disadvantage. He's a bigger target. Andre would be a MUCH bigger target.
Not that they were using firearms in the Princess Bride.