From The Descent of Man, Chapter 7, found here:
But the most weighty of all the arguments against treating the races of man as distinct species, is that they graduate into each other, independently in many cases, as far as we can judge, of their having intercrossed. Man has been studied more carefully than any other animal, and yet there is the greatest possible diversity amongst capable judges whether he should be classed as a single species or race, or as two (Virey), as three (Jacquinot), as four (Kant), five (Blumenbach), six (Buffon), seven (Hunter), eight (Agassiz), eleven (Pickering), fifteen (Bory de St-Vincent), sixteen (Desmoulins), twenty-two (Morton), sixty (Crawfurd), or as sixty-three, according to Burke. This diversity of judgment does not prove that the races ought not to be ranked as species, but it shews that they graduate into each other, and that it is hardly possible to discover clear distinctive characters between them.
[snip]
The question whether mankind consists of one or several species has of late years been much discussed by anthropologists, who are divided into the two schools of monogenists and polygenists. Those who do not admit the principle of evolution, must look at species as separate creations, or in some manner as distinct entities; and they must decide what forms of man they will consider as species by the analogy of the method commonly pursued in ranking other organic beings as species. But it is a hopeless endeavour to decide this point, until some definition of the term "species" is generally accepted; and the definition must not include an indeterminate element such as an act of creation. We might as well attempt without any definition to decide whether a certain number of houses should be called a village, town, or city.
[snip]
Those naturalists, on the other hand, who admit the principle of evolution, and this is now admitted by the majority of rising men, will feel no doubt that all the races of man are descended from a single primitive stock; whether or not they may think fit to designate the races as distinct species, for the sake of expressing their amount of difference.