And we let God decide who is unworthy. That job is not for us.
When someone is in an objectively evident state of mortal sin, the Church has not just a right but a duty to warn them away from full participation. This has been upheld from the earliest days and the interpretation of “judge not” you are working from is a modernist innovation.
God judges the ultimate destination of the soul but those who will not listen to the admonishments of the Church will be lost.