Until 1924, American Indians were not considered citizens of the United States, even though all were born within its limits. The reason for the exclusion was that they were considered to have their primary loyalty to the tribal nation to which they were born. As with foreign nations, the United States, as well as states like North Carolina and New York, engaged in treaties with these nations. In that sense, the Indian nations were as much foreign entities as Britain or Mexico. The purpose of granting American Indians citizenship was to facilitate their integration into mainstream American life.
If people here illegally hold citizenship in a foreign country, they are, like the Indians, considered to have their primary allegiance to that country, not the United States. Therefore the child is no more a citizen than the child of two Apaches born in 1890.
This is what Senator Howard meant when he excluded "persons born the United States who are foreigners, aliens...".