Also if my memory is correct, Sarah had ordered that Hagar and Ishmael leave because Ishmael was taking advantage of little Issac. Genisis also records that Sarah ordered Abraham to remove both Hagar and Ishmael. The present conflicts between Jews and Christians and Muslims began with Sarah and Hagar.
...and Abraham for taking Hagar and not waiting until the appointed time for THE son of the promise to be born. He was the head of the household and the one on good terms with God, he could have told Sarah "no" when she told him to have a child with Hagar.