What we do have is a right to arms for self defense. Hence his cleverly worded "except in private self defense" that you so casually glossed over.
The law isn't limited to robbery, rape and theft. People couldn't carry arms within the town limits, if that was the law there. Or carry concealed arms, if that was the law. Or discharge arms within town limits, if that was the law. Slaves could not bear arms, if that was the state or local law.
The exercise of police powers to regulate the ownership and use of firearms belonged to the states and their municipalities, as opposed to your dream of centralized government and rule from the federal bench.
Unlike you, the Founding Fathers were not dogmatic simpletons.