Sometimes it looks like he's about to smile when he says it. The gag reel is probably full of failed DAMMITS.
Oh, how fun it would be to SEE that gag reel! I bet indeed there are lots of extra, smiling Dammits.