Butting in here, sorry.
I am considering homeschooling my boys, and thought of this problem myself. I am an admitted failure with math. However, I discussed it with my husband, and he's agreed that he would help me teach the boys Calc, physics, ect if it came to that. English and History are my strong points and interests, so that would be my purview. If it was unworkable for some reason, there are educational co-ops that can help with the short falls. Plus, I have an advantage in that my children have many years to go before they are ready to tackle that kind of material, so I have time to educate MYSELF in order to teach it to them.
It isn't insurmountable, if you are willing and/or able to put forth the time and effort required. Sometimes it's not possible, but otherwise...why not go for it?