Depends on who Ismael married. If he married a Semite, his kids would be 3/4 Semitic, if he married a Hamite, they’d be 3/4 Hamitic, and who knows what other peoples he might have come into contact with.
Genesis 21:21 does say that Hagar found him a wife from Egypt, who are Hamites.
Guess I can’t argue with that logic.