That's exactly what happened. God said He would be a God to Isaac but never once said He would be a God to Ishmael. The Arabs are descendent of Ishmael. The Jews/Isrealites are descendents of Isaac. Abraham pleaded with God that Ishmael would live before God but God essentially said no.
Genesis 17:18 And Abraham said unto God, O that Ishmael might live before thee! 19 And God said, Sarah thy wife shall bear thee a son indeed; and thou shalt call his name Isaac: and I will establish my covenant with him for an everlasting covenant, and with his seed after him.
Jewish official prayers start by invoking YHVH, “the God of Abraham”, “the God of Isaac” (as opposed to the God of Ishamael), and “the God of Jacob” (as opposed to the God of Esau).