Difficult??? If anything, Cruzers would want to unseat Ryan more, because he blocked in the House everything Cruz was working for in the Senate--and Cruz even worked to sabotage both Boehner and Ryan in the House by trying to organize principles conservatives against him.
This is not meant as a putdown of Trumpsters, who have their own reasons to support the ouster of Ryan, but rather to point out one of many things both sides of the FR aisle should agree on with great ease.
But just not in the way they imagined. You bet your ass that we Trumpsters and the CruzCrew can unite against these jackasses.