There is a finite number of votes cast. There is also a finite number of potential votes. Most people who could vote do not. You are correct in that many people choose not to vote.There are a lot more non votes than votes and with those, the Dems could make the same argument.
For about all of my life I did not understand why so many did not vote. I have never missed an election at any level. Even in a small town election where I had lived only a short time and did not know the candidates or the issues well enough to believe I could make a good decision I went to the poll and got my ballot and just blanked it out.
After all these years I am beginning to understand why at least some people choose not to vote. I don't know what I'll do...probably vote....but there is no way I'll vote for somebody just because he's got the (R) or against somebody just because he's got the (D).