I’m figuring that both of the houses had doors, and both of the doors had locks, and that the keys to one house did not fit the locks on the other.
In other words, if it was an honest misunderstanding on his part, he would have noticed when the keys didn’t work.
Assuming the agent showed him #4 and gave him keys to #4 he still passed papers on #5.