You've got it; if we do nothing, they get to commit fraud; if we stop them, they sue on the grounds that we're "disenfranchising" them.
Yet, how many millions of dead Republicans were disenfranchised in the 2000 election by not being fraudulently called upon to send in absentee ballots, I ask you?