There's more to it than that. We're talking about segregated schools, poll taxes, and lyching.
You do have a point though. A person ought to able to serve who they want to serve. But that's not the platform Thurmond ran on. He thought segregation was just marvelous.
We are also talking about 6 trillion bucks spent on 13% of the population since 1965 and that population is STILL segregated, undereducated and the poorest group in the country. There has to come a point when the simple question of "why?" is asked of that group and "why do they vote 97% democrat?