DH and I nearly split up over this as well. We're 12 years apart (me younger) and when we first got together (I was 28) he wanted them and I wasn't ready yet. Then there was 10 years of dealing with his newfound substance abuse problem (long LONG story), then when I finally got him cleaned up and wanted kids he didn't anymore. With him at age 50 I can't say I could blame him.
The "selfish" label makes me want to throw up. I intentionally didn't bring any into a marriage that was in real rough shape, with one partner who was baked out of his mind 90% of the time, on the brink of divorce for 10 years, and I'M SELFISH? Bollocks. I did a kid a big favor by not dragging them into that situation.
I've (mostly) made peace with the situation - sometimes in life you get the present, sometimes you get the box, but the continued chastisement from the "you're selfish" crowd can be painful beyond belief.
LQ
I am glad you've made the best peace possible with your situation.