There's a little more to it than that. I'm no fan of the "civil rights movement," but the Birch Society was always a coalition of radical libertarians and traditional white Southern segregationists, and the latter very much considered "race mixing" to be the essence of Communism. Each side of this coalition was given free reign and for the most part ignored each other's "eccentricities." At any rate, white racialism was definitely a factor, along with the Communists in the civil rights movement.
Another factor was that, like Goldwater, the non-racialists in the Society considered all federal involvement in civil rights issues to be illegitimate and contrary to true Constitutional federalism. I pretty much agree with this position, but unfortunately the racialists in the organization (and in the conservative movement) simply refused to let their "blood is thicker than water" concept go. It's a darn shame that these issues could not have been addressed at local and state levels. It was a case of an unstoppable force meeting an immovable object.
ZC: that was a great summation; thanks for sharing.