Then what word is the object of "of" in that sentence, if not the "who"? If it's an object, it should be "whom".
I agree:"Of whom do they remind you?"