Anti-Semitism is a term coined by German anti-Semite Wilhelm Marr. Whatever its semantic origins, it specifically means Jew-hatred.
Semitic people encompass people in the region of Jewish, Christian, Muslim and probably some who believe in Zoroasterism
Anti-Semitic would have to include all those people. Whomever coined the term simply narrowed it down for their own nefarious purposely but the term would have to encompass all the above to be accurate.