Of course it is not only the Muslims that cause trouble. Haven’t some rabbis decreed that no Jew should go up to the Temple Mount?
There are differences of opinion in rabbinical circles. Everyone holds that one cannot enter the holy of holies, but that is on the large rock presently occupied by a mosque with a golden dome, so it’s pretty easy to avoid.