He had every right to tell them to leave.
Did he have the right to ORDER them to leave? The answer is unclear, and almost certainly "no."
Did he have the right, under any circumstances, to commit an armed robbery?
How many times do I have to repeat that he was stupid for pulling a gun, he absolutely every right to ORDER them to leave.
It’s possible the people were not violating the rules by fishing, if the rules prohibit fishing, it’s incumbent to for the owner to tell them to leave, order them to leave or whatever you want to call it.