“...his distant relations were racist...” I would argue that being a slaveholder does not equate to racist, but whatever makes Ben feel and look bad (his acting comes to mind) is okay by me.
No need to bring race into it when "evil" will suffice as a description.