Non-citizens are guests in this country and we have every right to remove them for whatever reason we choose, even as we have the right to kick guests out of our homes for whatever reason we choose. Chances are we wouldn't throw a guest out of our homes without good reason. I dare say that if I found a guest in my home to be practicing Islam, that would give me enough cause for concern that I would not want them in my home and more than I'd want a Satan worshipper in my home. In my opinion, there is little difference.
There are provisions for this. (like an anti-anarchist provision from the 20's for example) However, you can't do it arbitrarily. There is a judicial review.