The Bayonet Party can easily get him out of office in the general, if they vote for the Democrat.
His problem will be the primary if people are annoyed at him, not the general