On the other hand, "racist" implies that a separation is necessary because one race is inferior or superior to another, a qualification that is absent in "segregationist."
Your use of the term "segregation" seems incorrect. You seem to be referring solely to the relatively recent phenomenon -- mostly seen among black students on college campuses -- often referred to as "self-segregation." That's like using the term "discrimination" to refer solely to reverse discrimination/affirmative action.
Segregation is generally understood to refer to the legally-enforced separation of the races as under the South African apartheid system or the Jim Crow laws in the old South. Segregation is neither racially neutral nor in any way related to free association and individual liberty.
For example:
"All segregation statutes are unjust because segregation distorts the soul and damages the personality. It gives the segregator a false sense of superiority and the segregated a false sense of inferiority. Segregation, to use the terminology of the Jewish philosopher Martin Buber, substitutes an "I-it" relationship for an "I-thou" relationship and ends up relegating persons to the status of things. Hence segregation is not only politically, economically and sociologically unsound, it is morally wrong and sinful. Paul Tillich has said that sin is separation. Is not segregation an existential expression of man's tragic separation, his awful estrangement, his terrible sinfulness? Thus is it that I can urge men to obey the 1954 decision of the Supreme Court, for it is morally right; and I can urge them to disobey segregation ordinances, for they are morally wrong."
Martin Luther King, Letter from a Birmingham Jail.
Segregation is not the product of free association, but of its antithesis. Segregation did not come about because people freely decided to associate solely with members of their own race, but because a series of laws were expressly enacted to prevent people from freely associating with members of a different race. Apartheid systems and Jim Crow laws would be superfluous if segregation came about from free association as you imply.
Your suggestion that segregation is not in and of itself racist is eerily similar to the U.S. Supreme Court's reasoning in its long-discredited ruling in Plessy v. Ferguson from the 1890s. Plessy upheld the constitutionality a state law requiring separate railroad carriages for whites and blacks. The majority ruled that "separate but equal" facilities did not violate the 13th or 14th Amendments. In the Court's view, the law separating the two races "did not stamp the colored race with a badge or inferiority" and if such perception existed, it was not contained in the law, but in the plaintiff's decision to put such a construction upon it.
In a famous dissent, Justice Harlan pointed out that separate but equal facilities were separate in fact, but equal only in theory. In practice, separate but equal facilities were everywhere and always separate and unequal. Harlan predicted that the Plessy decision would someday be deemed as pernicious and discredited as the Court's decision in Dred Scott. Needless to say, the bloody legacy of Jim Crow and the civil rights struggles have proved Harlan right and the Plessy majority wrong about segregation.
If we assume your comments refer only to self-segregation, then I agree that it should not be the government's business if people wish to self-segregate. But I don't see how such people's attitudes could be motivated by anything other than racism or ignorance. Self-segregation is undoubtedly less odious than segregation under apartheid or Jim Crow (because of freedom vs. coercion/violence). But it is still odious to a lesser extent.