Plus, I wonder what it must have been liked for the hobbits to be consistently dismissed just because of your size. It happens more to Merry more the others... (yet Gimli is never dismissed because of his size, but then again men had more dealings with dwarves then hobbits) but Sam and Frodo must feel it in this company of men. Hobbits are every bit as courageous and honorable and smart as the men of Gondor, but because of their size, they are diminished in the eyes of the men.
If it wasn't for the fact of Sam's size, I wonder how the men would have reacted to Sam's little tirade. (And Sam was right... who were these men to question Frodo so?)