Maybe in the past, he wandered over, hoping for some actual or vicarious pleasure, by getting involved with the swingers.
Instead, he finds locked doors to the garage, but wide open access to the house. He wanders around, undetected. He learns the route, that way.
In fact, if the timeline on the night of the abduction allowed, this could explain his entry. Retrace steps for exit.
I regret bringing this up. I only stated it to demonstrate why speculation should not become the basis of reasonable doubt (as stated in the jury instructions to CA jury approved by SC
It reminds of the questions that a child poses along lines of why is the sky blue ?