Because, in addition to Goliath, there were his four giant brothers.
Right, one stone for each. These were the 5 remaining giants in the land, offspring of fallen angels and the daughters of man...