It has been said these buildings were close and that he moved from building to building. It is possible we are looking at the wrong ladder that he climbed up.
Possibly.