Currently, because the people of Gibraltar voted to stay part of the British Empire.
By your “logic,” Britain should rule the United States.