He is only basically considered to be speaking infallibly when declaring/defining a matter on faith and morals to the whole church. But many Rs also hold that the pope has no authority over them in the manner of teachings which prior papal teaching said they were to submit to. Many RCs do not even consider Francis (whom they usually refer as Bergoglio) to be a valid pope.
Here Francis has been judged to be a Marxist, dictatorial, power hungry, false prophet Pope, and in Nostra Aetate and Evangelli Gaudium to be teaching (formal I assume) heresy, and that a heretic cannot be a valid pope, and that Pope Benedict failed in his duty to the Church.
However, considering that it has been acknowledged that Francis was validly elected as pope, that makes him the valid pope of the Roman Catholic church whether they like him or not.
Because, like it or not, their church leadership, which has the authority to do so, decided on it, not them, and therefore, they have nothing to say about it.
Their duty is to submit to the pope, as stated MANY times in the history of the church, and do what he says.