The goat was chosen as a symbol long ago by people who worship Satan, because the Bible says that God is going to separate the sheep from the goats. By representing themselves with a goat they are making the point that they don't follow God. In addition, Satanists refer to the "left hand path" as their lifestyle, in reference to the fact that Christians are told in the Bible that Jesus sits at God the Father's right hand in heaven.