Hmmm...you sound an awful lot like the Arabs who claim that there was never a temple on the Temple Mount in Jerusalem. Muslims are notorious for destroying and supplanting other religious shrines all around the world, and then eliminating evidence of the prior sites.
Still, that didn't mean the mob had a clear title to the site. Land is precious in India, and someone else may have a better claim.
In any case, after reading as much as possible about what the findings of the archaeologists are regarding the temple, about all we know is that it was done in Indian style, and that could involve MANY DIFFERENT GROUPS, and even if Hindu, it could still involve MANY DIFFERENT GROUPS OF HINDUS.
My money is on it having served as a Buddhist or Jain temple as its last use before acquisition by the Moslems.