I did not see that "small man" as a reference to his height. Is he short?
A small man in this context to me refers more to his total lack of character.
``You are a very small man, and like other men (of small stature), you have a Napoleonic complex and a need to compensate for what you lack,'' Hawthorne said Monday in court. ``Small man, big bomb.''
Strangely enough, I am half an inch shorter than this guy, and I've never been called 'small' out of anything other than an ironic context. Maybe it's how he carries himself.