Infallibility belongs in a special way to the pope as head of the bishops (Matt. 16:17-19; John 21:15-17). As Vatican II remarked, it is a charism the pope "enjoys in virtue of his office, when, as the supreme shepherd and teacher of all the faithful, who confirms his brethren in their faith (Luke 22:32), he proclaims by a definitive act some doctrine of faith or morals. Therefore his definitions, of themselves, and not from the consent of the Church, are justly held irreformable, for they are pronounced with the assistance of the Holy Spirit, an assistance promised to him in blessed Peter." --Papal Infallibility - at Catholic Answers
To the mind of the general reader, mention of the Vatican Council will usually recall two facts: that it was the occasion of the definition of the doctrine popularly called Papal Infallibility, and that many of the bishops present were opposed to the definition. The more erudite will add that their opposition was not to the doctrine itself, but to the policy of choosing this present moment to proclaim it--the definition, they would say, was held not "opportune." ... The leading critics of the "definition policy" were French, German, and Hungarian --CHAPTER 20
Following the first Vatican Council, 1870, a dissent, mostly among German, Austrian and Swiss Catholics, arose over the definition of Papal Infallibility. The dissenters, holding the General Councils of the Church infallible, were unwilling to accept the dogma of Papal Infallibility. Many of these Catholics formed independent communities which became known as the Old Catholic Church.
At the First Vatican Council he was one of the most notable opponents of papal infallibility, and distinguished himself as a speaker. The pope praised Strossmayer's "remarkably good Latin." A speech in which he defended Protestantism made a great sensation. Afterwards another speech, delivered apparently on 2 June, 1870, was imputed to him. It is full of heresies and denies not only infallibility but also the primacy of the pope. The forger is said to have been a former Augustinian, a Mexican named Dr. José Agustín de Escudero. --Joseph Georg Strossmayer, the Catholic Encyclopedia, 1913
But it's important to understand that the idea pertains not to the person but to the office. We are not saying that Popes are not sinners or are not afflicted with the same injuries that afflict all sinners, faithful or not.
We are saying that while God lets us as individuals mess up incredibly awfully, he still provides us with reliable testimony to His truth. The doctrine, from the point of view of the Catholic, is about the love of God for his poor confused people, not about some guy in Rome.