“If that were true then the child of any foreign national who happened to be born on the base would be a U.S. citizen.”
It only applies to American military CONNECTED to that base. Even if the military lives off the base, they are still connected to that base and retain their American citizenship.
Because the offspring get their citizenship due to the citizenship of their parents. Being born on a military base in a foreign country has no bering on that because U.S. military bases are not U.S. territory.