They’re in different buildings. Library is in ground floor of residence; Oval Office is first floor of West Wing (but still on same level).
You have go from far corner of the Residence down the Center hall, go through Palm Room, along the West Colonnade past the Rose Garden (and press corps offices and briefing room) and then enter the West Wing and take another hallway to the left to the Oval Office, or continue on outside walkway to enter through the French doors.
Thanks.