You always need to check out the paternities. With a far higher maternal death rate back in the 1800s (and before) it wasn't unusual for a well-situated man to have several wives in a relatively short period, with the true identity of the birth-mother getting lost in popular view.
Still, the family may have known something that was never made public.