{He has a right to set his own agenda.}
Yes, and others have a right to assess and criticize.
As far as I know (not being Catholic) he is only considered infallible (not by Protestants) in matters of religion.
The social agenda he promotes is antithetical to Christianity.
More than that, as Vatican I defined: "when, in the exercise of his office as shepherd and teacher of all Christians, in virtue of his supreme apostolic authority, he defines a doctrine concerning faith or morals to be held by the whole church".
In other words, it doesn't necessarily cover his private communications on religion. The Council Fathers who wrote this definition had, of course, the example of Pope Honorius I before them, who was slapped after his death by an Ecumenical Council for writing a weaselly letter to the Monothelite heretic Sergius.
Why did the Pope meet with prisoners in the U.S., but not with prisoners in Cuba?