if you are dating someone, in the 'in-your-face' spirit of this web site, i would tell them exactly how you feel about their question, with appropriate portions of tact.
if you are living together -- i suggest that the reason the question is irksome is because you are feeling guilt. in this case the situation is under your control
if you already married and you reveal this fact, prepare yourself for the question, "when are you going to have kids"
See my post #30