Because creationists make the rest of us conservatives look bad and turn people away from the conservative movement.
To you only, maybe. Creationists are the majority of the conservative party, so I believe it attracts and retains, not scare people away.