Destroying Germany required the Nazis be driven out of France and other countries. It would have been next to impossible to successfully invade the northern coast of Germany in order to destroy Germany while leaving France untouched.
And at the time of the Normandy invasion, the ability to build and drop an atomic bomb on the enemy's homeland was over a year away.
I’m not disagreeing with your point. Destruction of Nazi Germany was the goal. Norway, Denmark, most of the Netherlands, the Channel Islands, and French port cities were bypassed despite the suffering the people of those places endured right until the surrender of Germany.