In a nutshell, the story goes that Abraham's father was idol maker. And one day, one of his carved idols fell into the fire and burned up. Abraham was aghast thinking how could this be a "god" when it is consumed by fire.
His dad wasn't moved. He said that was the way the "god" gave him warmth and allowed him to prepare his meat.
But Abraham, as the story goes, fell to his knees and asked God in prayer, if you really exist please reveal yourself to me.
I think God had other plans for Abraham and kept him separate from the service of “other gods”. I think that was the point of verse 3 and the contrast between those who came before Abraham and Abraham himself. Abraham was set apart and thus I don’t believe he ever was an “idolator” in the sense that he served other gods.