Let me reiterate that infallibility is not a positive power of the Pope; it is a negative protection of the Church. Protection from what or from whom? Protection from the errors of popes!
It is a guarantee that, whackadoo as an individual pope may sometimes be in his activities and his opinions, he shall not have the power to impose them on the Church and thus draw the Church into error. (If he were able to draw the whole Church into his mistakes, that would be the "gates of hell" prevailing against the Church, which Our Lord promises will never happen.)
The conditions required for infallible papal teaching are as follows (from the First Vatican Council, 1870):
For a teaching by a pope or ecumenical council to be recognized as infallible, it must be:
That last condition is fulfilled by words such as "By the authority of Our Lord Jesus Christ and of the Blessed Apostles Peter and Paul, and by Our own authority, We declare, pronounce and define the doctrine . . . to be revealed by God and as such to be firmly and immutably held by all the faithful."
So the only time that a Pope has to be thought of as perfect, is when he is passing on to all Catholics, what God just told him, or that someone we knew, is now with God and is a saint.