Dwarves are a people group, they are not sufferers of “dwarfism”.
He is equating his physical malady with their unreal existence as a fictional group. Tolkien would be horrified and flabbergasted at such an equation.
Chip the glasses, crack the plates, That’s what Bilbo Baggins hates!
Since you brought up Tolkien, and dwarves.