So the racial restrictions were fine with you?
Kinda sounds that way, doesn't it?
That said, it is his land, he can sell (or not sell) it to whomever he pleases. That right is afforded to everyone not nailed by emminent domain, regardless of race, creed, etc..
If there was a covenant in the deed, chances are it was there when he acquired the property, and the poor slob is just stuck in the middle.
I believe such covenants are deemed void, anyway, so he just gets nailed every which way for something someone else wrote in, and that he mentioned as a caveat.
If you don't think that is a raw deal, then you should reevaluate what you think 'fair' is.
Think of it as "affirmative action" for white people.