My impression is that everybody thought it had been left aboard the lunar lander, which eventually crashed on the moon surface after the mission. From the collection of stuff, it looks like Neil basically grabbed everything that wasn’t “nailed down” and didn’t functionally *have* to be left behind.
Aside from the historic camera, I don’t have any problem with someone who risked his life keeping a few “mementos of the trip” around. He was entitled to have his own “swag bag” IMO.