There are women who can’t conceive, and I know of at least one marriage that ended because he wanted kids and never got any, so he left, remarried, and then had kids. Although “impotent” does connote male failure-to-perform.
"Impotent" denotes male inability to perform; that is the word's definition.
If the lawmakers had meant to say that annulment was available in cases of sterility, barrenness, infertility, etc., they had those words available. Perhaps such things featured in divorce statutes, but that was not the topic.