I hated it for the same reason: in that book he engaged in more than a little a priori assuming himself.
Specifically, the primary intent of the book was to debunk the means by which people meaasured "intelligence differences" between races. It was useful for debunking those particular tests; however, Gould's clear intent was for us to conclude that there were no intelligence differences at all.
The problem is that in order to be able to say this, one must also say that the evolutionary adaptations that produced distinct and measurable physical differences between races, could not extend to intelligence, despite the fact that he considered intelligence to be an evolved characteristic. This is clearly a flawed assumption -- there are obvious trades between, say, intelligence and speed, or eyesight, or even just the ease of gathering food.
Gould was a highly political animal, and it often showed up in his scientific writings.