Probably because he was afraid of getting caught. The shooting took place right in front of the building (which I have visited), so it would have been thoroughly searched, and the weapon would have been found.
Oswald was seen on the second floor having a coke. The two women never saw him pass them sitting on the staircase. Also the other day, the doctor in that attended JFK in the emergency room testified to Congress that Kennedy had two entrance wounds from the front. He stated he and a fellow doctor inserted a trach tube into the throat entrance wound. He stated JFK had a frontal wound at the hairline at the right temple. In essence Kennedy was struck from the front twice.