There was also the military vs civilian factor, Armstrong being a civilian.
According to Deak Slayton, head of the Astronaut corps, Grissom would have been his choice to be the first to walk on the Moon.