As you say it is a conundrum. However, the answer is indeterminate as it is not clear whether the man he met was also going to St. Ives or somewhere else. Perhaps the term “met” can be interpreted that they were going in opposite directions but, again, maybe not.
That is true also. But, it isn’t a math problem. It is a logic problem.
Which makes it a problem in semantics.
In the same class as the riddle “If a tree falls in the forest and there’s nobody to hear it, does it make a sound?”