He was also engaged in a major war with France. A far more powerful opponent then the Colonies. He didn’t seem to have an issue getting lots of soldiers and sailors killed by the French.
People take actual threats very seriously. The colonies were never a threat to Britain.