Lots of people flew the X-15. From the Wik:
“During the X-15 program, 13 different flights by eight pilots met the USAF spaceflight criterion by exceeding the altitude of 50 miles (80 km) thus qualifying the pilots for astronaut status. The USAF pilots qualified for USAF astronaut wings, while the civilian pilots were awarded NASA astronaut wings in 2005, 35 years after the last X-15 flight.”
Elsewhere it indicates that 12 different pilots had X-15 experience.
Is there a list of names?