That is not true.
Sarah and Abraham were married. Hagar never was the WIFE of Abraham, hence Isaac is the LEGITMATE son - between tow married people. NEVER is Hagar referred to as Abrahams's wife because she is not - she is just a maid to Sarah.
The children were Abram's, and that's all that counted in that society at that time. It is inappropriate to read modern understandings into events in his life.