I think he was a savvy businessman and when I used the word “slumlord” then it was to describe the society’s perception of his business of running lower income residential housing, even if done competently, being low prestige.
I can understand that use to denigrate somebody to create an image.