The only right answer in either case is, "How could it possibly be any of your business?" :)
It's even worse when I am asked if my children have different fathers. They only look like little clones of him. Those who look any any different *gasp* look like me. I have thought of telling anyone who asks the father question, "They have the same father. We just don't know who their mother is."