And the so called rhythm method is in reality another form of contraception.
An excellent question. I recommend to you the excellent article Is Natural Family Planning a 'Heresy'?.
You cannot "rob" a man with no money, any more than you can "strip" him if he is already naked. Likewise, since conjugal acts carried out precisely in the infertile period do not, by the very nature of the case, have any natural procreative potential to begin with, it is obvious that they cannot be "deprived" or "robbed" of that potential.
Any intentional and planned act that prevents contraception in a conjugal relationionship contraception. It takes a special mind to deny this.
But that is how the Catholic Church approaches the isse of divorce as well -- a denial. Officially, there is no divorce, but there is "annullment" -- the Church says that the marriage never took place (in a sense that it was an invalid Sacramanet). But, there is no limit to the number of annullments! The Orthodox Church, on the other hand, recognizes that we are human and fallible, and will grants us two chances to marry and divorce and no more.