What would hurt them far, far more, is the handful of Blue Dogs and maybe 20 swing-state moderates either voting against impeachment or abstaining.
Anything less than a party-line vote would give Trump a chance of turning the House.
The will have the votes for articles of impeachment. They will next have the media harping that Trump can’t nominate judges if under impeachment. Look for Rommney and his friends to say, “yep, that’s right.”