their relationship is going backwards - sounds strange but consider the show. she is moving backwards in time and he forward - their relationship was also featured in earlier series with other men playing the Doctor
Well, he moves around in time, but he does have a history and a future. The library was the first time he met her, but not the first time she met him.