God bless you. It will get better. Once you've made it clear that you're disinterested in the opinions of the naysayers, they'll leave you alone. I don't think it's rude at all to tell a busy-body to mind his/her own business. In time they may become your biggest supporters.
I speak from experience. My mother was incredulous when she learned that we would be homeschooling our son. I have great respect for my mother. She is a saint. But it is the responsibility of my wife and myself to raise our son. In the nicest way possible, I reminded her of this. She held her tongue; in time she saw how well this was working, and now she is a strong advocate for homeschooling. Now, she has said that she wishes my sister would homeschool her son, and I've had to remind her again, in the nicest way possible, to mind her own business.