I think my great grandfather was an atheist, and have often wondered if that had anything to do with anything.
Statistically, I doubt that is a greater number than the law of averages
My paternal grandmother's grandparents (my gr gr grandparents) were first cousins, all the male line bearing the paternal surname has died out in the descendants, many died young and/or unmarried, but one daughter lived to be 89. It's kind of pointless to speculate, and not a good idea to blame your own problems on those who went before when they could be a genetic accident or something you brought upon yourself. On the upside, my grandparents' three children had extremely high IQ's which was passed down to some extent to my sister and me, nature vs. nurture? Somebody did a study, and the male line of Martin Luther died out, thought it was the 17th generation but could have been sooner. That subject fascinates me, how some families always have males to carry on the family name.
Any risk from inbreeding is only to the children of related parents. If your mother and father are not related, then all “inbreeding” in previous generations is neutralized.
Relax.