That is your religious belief.
Those folks who were praying were exercising their religious beliefs. “Free exercise” does not mean we get to impose our beliefs on others. They are FREE to have and exercise their own.
It stands as a matter of fact apart from any religious belief that most speech brings consequences and prayer even more so. Let the college prosecute those who pray in “free speech zones” (a stupid idea if ever there were one) and see where it gets them. Let a man pray in public and see what it gets him as well. One thing that will not happen, however, is nothing. What exactly is meant by the word “impose?” I cannot post whatever I wish in this forum without regard to rules and consequences. Does that mean “free speech” is not allowed here? Yes. Does it mean I am being “imposed upon?” Absolutely. We are free to speak and to post, yes, but not without constraint.