Incidentally, I'm not sure why a statue of Calhoun would be put in Harlem. Did he have some tie to the area?
I can understand why a statue of Lincoln might be put in Richmond, though. It's because Lincoln was President of the United States, and Richmond is a city in the United States. Moreover, the statue is to commemorate a trip Lincoln actually took to Richmond.
I'm sure you can see where your analogy fails.
There's a statue to Mahatma Gandhi in Union Square - and he has a much more tenuous link than Calhoun.