I think you’re right. There’s no reason for him to hang around because he wants to find his friends.
Daryl actually trapped Jim’s group, insinuated himself into it to take them all out and then get back to finding everybody else. He sat himself down in the road all ‘boo hoo’ to attract them to him.