Actually, in this case, they are. Customs & Border Protection (of which the Office of Border Patrol is a part) has the right to inspect any person or conveyance at or near the border. A country has the right to inspect anything crossing its borders, for no other reason than the fact that it's crossing the border. That's been the standard US law and international law for literally hundreds of years.
The inland searches are a little more complex, but not much. Let's say you have a swath of border a hundred miles wide with only one decent highway heading north out of it. Common sense dictates placing a checkpoint on that highway. You don't want it too close to the border, or it will simply be bypassed. So, you end up with checkpoints 50, 75 miles inside the US border.
Bottom line, OBP acted within their authority. The good Pastor went out with the intent of starting a confrontation, and he got his wish.
They were fifty miles from the border. He was not crossing the border; only traveling within the US as an American citizen.