Wealth did not destroy his family - attitude did. It is all about values he did not have to pass on to his children.
Also, it reflects the poor jobs their mothers did in raising them. Jackie O was probably the only mother of that generational clan that did a good job raising her kids.