Then the question becomes:
Which of you is led by the Holy Spirit?
Where do you get the idea that everything a person says or does is led by the Holy Spirit? The Church teaches that the Holy Spirit protects the Pope from formally teaching an error on faith or morals, it doesn’t teach that everything a pope or a person says or does is guided by the Holy Spirit. There’s the Voice of God, the voice of the flesh and the voice of the devil sometimes disguised as an angel of light. There are our own sins which we can mistake as the Voice of God.
What papal infallibility entails and doesn’t entail tends to often be poorly understood here.