But you just said "The Catholic Church does not teach that it is morally wrong to prevent pregnancy"
Now your saying it's wrong to use a condom...
Am I missing something?
Yes. You've not considered the distinction between means and ends. Even if you have a perfectly legitimate end (avoiding pregnancy because of a serious problem) you still have to use good means.
Remember that, unlike animals, humans are created "in the image and likeness of God." That's not just a bit of cheap self-flattery ("Woohoo, lookit me, I'm so godlike") but a mandate to respect the "human form divine" (as the poet Blake put it.) Briefly, you can aim to correct and perfect us human beings, but you shouldn't aim to re-engineer or redesign us. "Deconstruction" of humans is morally objectionable.
With this in mind, you can see why using a deconstructive means (like, for instance, castration) would be morally wrong, even if your aim is to avoid pregnancy. The main problem with contraception is that it treats fertility as if it were a disease or at least a design flaw. Contraception aims not to perfect the human sexual design but to extirpate parts of it. Contraception deconstructs natural sex.
Interestingly, part of the human sexual design is that we females are only intermittently fertile. That's why using a little forethought about the timing of intercourse can put you in a good position to either achieve or avoid pregnancy.
The whole aspect of the sacredness of the created human design is so seldom spoken about, or even thought about, that it seems almost culturally inaccessible to us. We're more likely to think of our sexual design --- or even human life itself --- as being literally insignificant and inconsequential. Perhaps it seems that way to you.
If that's the case, it would be easy to blow off the whole discussion at this point. However, the Catholic view --- if you sincerely want to understand it --- is very much based on the idea that our sexual nature has a sacred aspect.
I think this article is worth looking into:
http://www.touchstonemag.com/archives/article.php?id=17-01-020-v
Thanks for reading this far. Thanks for your patience.