Talk to a lawyer before you do anything.
If you say anything to the neighbor, they might launch legal proceedings. It is possible if you quietly make alterations to things, without him knowing, he will not be able to claim you allowed the pipe without issues, and you can claim you were still asserting control over the property.
Don’t make a move if you do not know how it will play out legally.
Agree. It is about Adverse Possession. Your neighbor may not know about the legalities. If you can afford a consultation with a lawyer there, I would suggest it.