Not exactly, it's because you are spreading your multiple genes into a confined area increasing the odds of inbreeding.
Take a not so hypothetical, a man has 4 children with 4 women. What are the odds of half siblings mating, especially if the children don't know their father?
A few more generations and the odds get really bad.
Slim...
You are describing life in the underclass, where an alpha-thug may impregnate dozens of girls in a neighborhood, whose spawn may grow up and inter-breed with their own half-siblings.