Publically contradicting the Deposit of Faith is not just a job performance issue. It is much more serious than that. It calls into question his very Catholicity which calls into question his legitimacy as the leader of the Catholic Religion. Does Francis profess the Catholic Faith as it was promulgated for 1960 years? Does Francis profess the same Faith that the pre-Vatican II popes professed?
He was duly elected in accordance with Tradition. Holding fast to Tradition requires accepting that he is pope. It doesn't preclude us from believing that he's a bad pope.
It's not within the province of the the laity to declare a pope invalid.