As someone who spent some years in remote, sparsely populated areas I can tell you that this was usually the case. When law enforcement was hours away and often not available for anything but major crimes, ad hoc groups would form. I never saw any of the mob mentality that is depicted in movies but usually older and very somber and sober respected citizens would meet to consider the situation.
The deliberations were probably more intense than actual jury trials. Each man knew that he was putting his name and reputation on the line. There would be no legal games and no absolution before God if they were in error. They most often tended to the minimum practical solution to the problem, most often a warning that the transgressors crimes were known and that they would stop immediately or the criminals would face the wrath of the community. Usually that stopped the behavior.
If that didn't work, the subject would have a visit by a group of very serious and very fit men with a final warning as a demonstration of what they were facing. From there things would escalate rapidly. There would probably be a beating. Then there would be a notice that their presence was no longer desirable in the county. Then there would be a 24 hour deadline to leave town. Then...
Rarely did things go past steps one or two. It was efficient and effective and I never, ever, saw it abused.