An excellent question. I recommend to you the excellent article Is Natural Family Planning a 'Heresy'?.
You cannot "rob" a man with no money, any more than you can "strip" him if he is already naked. Likewise, since conjugal acts carried out precisely in the infertile period do not, by the very nature of the case, have any natural procreative potential to begin with, it is obvious that they cannot be "deprived" or "robbed" of that potential.
What a mastery of rationalization! The bottom line is (and from all this philosophical mush many catholics don't seem to be able to see the bottom) that having sex for pleasure (knowing there will be no conception) is not condoned by the Catholic Church -- and having sex knowing that you are shooting "blanks" or that the woman cannot get pregnant is having sex for no other reason than joy.