Hmmmmmmmm. We do KNOW that the vanDams had people in the house Friday night, yet the police did not find their fingerprints......It is possible for a person to be in a place and not leave behind evidence (as you know).
Maybe in the past, he wandered over, hoping for some actual or vicarious pleasure, by getting involved with the swingers.
Instead, he finds locked doors to the garage, but wide open access to the house. He wanders around, undetected. He learns the route, that way.
In fact, if the timeline on the night of the abduction allowed, this could explain his entry. Retrace steps for exit.
Highly unlikely in a 2 story house with the bedroom upstairs. IMO