You are entirely correct, at the very least in regards to correctly judging the "elect"ness of others. This is where the P in TULIP - the Perseverance of the Saints - comes into play as a visual aid to the observer. None of us - from the Pastor down to the layperson - knows what's in the hearts of that person in the pew next to us, or the church across the street. What we can do is observe their lives, and offer a temporal judgement as to their spiritual condition, tempered as limited to only as much as has been observed. When churches do this, people are either baptized, offered communion, or perhaps branded heretics or excommunicated. Having said a prayer found at the back of a tract may not be enough, at least not from the standpoint of the observer. It wasn't for Saul-turned-Paul. The church in Jerusalem wanted to take a wait-and-see attitude, based upon his past life and recent conversion, but it took a divine act experienced by the observers, to convince them otherwise. Paul's initial testimony did little to convince them.
Barring the witness of a divine act, none of these actions are IMO categorical, final decisions that render a person saved (or damned) eternally. Instead, they are a church's "best guess" as to that person's spiritual state are, and are recognitions (or denials) of temporal grace to that end. There is always a final judgement, and fallen men and women may misjudge (and mis-prosecute) others. But the more grounded in Scripture they are, and the more they are earnestly seeking the Lord's face in their decision-making, the less likely it will be.
I have no problem with thw 'P'. My comment was directed to unconditional election. No man can really know if unconditional election is true. One may speculate, but to claim it is absolutely true requires a great deal of faith.