It would be a major upset for all the reasons you stated. But, in a low turnout primary, ya never know. Might just be enough ticked of Trump supporters that show up and Cantorize him.
Happily, yet tragically, if Ryan wins his primary......I really believe that he will lose in November against the Dem......there are that many pissed off people in the district to tip the election to the Dem.