It goes far beyond “looking bad” for Mueller to remain as special prosecutor because he and Comey are joined at the hip. It is against the law that allows for a special prosecutor. If the Republicans are sporting anything but a pair of miniature marshmallows, collectively, they will force Mueller to resign.
Remember that a bunch of the RINO never-Trumpers in the GOP dislike Trump even more than Schmuckie. Doubt me? Look how they are busting ass to implement Trump's agenda.