Cantor lost because he diverged so far from his party’s platform (and got so out of touch with his district) that a political unknown could beat him simply by running on the party platform.
Amnesty is probably the most significant, but I imagine spending/debt and failure to serious address Obamacare added up to.