Yes, but doesn't God also make a pact with Ishmael as well? "As for Ishmael, I am heeding you: I will bless him. I will make him fertile and will multiply him exceedingly. He shall become the father of twelve cheiftains, and I will make of him a great nation." (Gen. 17:20)
He certainly did make a pact with Ishmael as noted in my original post. I believe the 12 princes referred to are the kings of the Arab countries. But, God also makes it very clear by following up His deal with Ishmael by clarifying His intent for the land to belong to Isaac when He says, "But my covenant will I establish with Isaac".
Also, in Gen. 21:10 Sarah says, "Cast out this bondswoman and her son: for the son of this bondswoman shall not be heir with my son, even with Isaac". And God agrees with Sarah in verse 12, "in all that Sarah has said unto thee, hearken to her voice; for in Isaac shall thy seed be called".