I don't recall a "right to drive" in the Constitution -- unlike the 2nd amendment.
The Constitution doesn't bestow the right to keep and bear arms either even though it is iterated in the Bill of Rights. Take a look at the Ninth Amendment for "rights not enumerated". Also read the Tenth. Then there is Article IV, Section 2: "The Citizens of each State shall be entitled to all Priveleges and Immunities of Citizens in the several States." Numerous people have sued the government, and won, to drive without lisences or vehicle registrations. The court found in their favor, why are they granted a Privelege or Immunity from the law that the rest of us are not?
Doesn't a Citizen have a right to travel freely within our own country? Is it the States' business how we travel? The current mode of common transportation is by automobile. To be denied the ability to travel freely is to severely hamper the freedom of the individual. It would be economically crippling to most people to be denied the ability to drive. Placing restrictions on that without just cause or due process is invasive upon the individual liberties of the Citizen.
You avoided the most important question; "If a driver has not demonstrated a danger to the community or caused harm to anyone why should he/she be tested, lisenced and registered by the government?"