The difference is intent. Infertile couples aren't infertile by choice.
There's also the category of people past the child-bearing age who get married. For some of these people, it's better for society, and for them, to get married rather than to become "dirty old men" (or women). As St. Paul said:
1 Corinthians 7:8-9Now to the unmarried and the widows I say: It is good for them to stay unmarried, as I am. But if they cannot control themselves, they should marry, for it is better to marry than to burn with passion.
I would submit to you that those couples who are "infertile by choice" are doing the world a favor and rather than being called names such as "selfish", they should be thanked for not bringing children into the world who would be treated as an inconvenience or even resented. The last thing this world needs is another neglected, resented child growing up to be a mass murderer or something. No child deserves to be raised like that.