Started from the start.
It used to be only delegates.
This popular vote stuff came later.
If we had a multi-party type of system, with coalition government like in many places in Europe, then maybe it would be less important to have each individual candidate selected by the popular vote. In our essentially two-party system, however, it disenfranchises the voters to have their only choices on the ballot two people who were chosen by unelected party officials.