All you have to do is intercept the boat, and turn it around. Do this 3-4 times, and they stop coming.
Or just sink one of them.