We have two things working against each other.
One, no question that historically the party in power loses seats in an off-year election.
Two, the midterm voter tends to be older, better informed, and more motivated.
Three, the midterm voter of the opposing party than the one in the White House is more motivated.