I think God had other plans for Abraham and kept him separate from the service of “other gods”. I think that was the point of verse 3 and the contrast between those who came before Abraham and Abraham himself. Abraham was set apart and thus I don’t believe he ever was an “idolator” in the sense that he served other gods.