It wasn't just the speech (though that was a big part of it). It was also the damage he did to Bush by challenging him in the primary.
Ridiculous
Most of Buchanan's supporters likely voted for Perot in the election. Yes, his challenge to Bush damaged GHW's re-election chances just as much, or more, than this speech.