You only say that because you have all of your fingers and toes.
No, I got my masters in Special Ed. I've met lots of folks with missing thumbs, hands, arms, or defective intelligence. It's sad, but over the long haul, those defects caused by genes rather than environmental factors, are discarded. There are bad recessive genes that maintain a relatively low level in the gene pool, but they cannot ever reach a high level because they interfere with survival and reproduction.
There are also defects like sickle-cell trait that have some selective advantage in tropical climates.
But genes do not degrade over time. Reproduction is digital, with layers and layers of protection against degradation.
But, of course, no protection against change that is neutral or advantageous to reproduction.