First, as you pointed out the French forces in North Africa were under the control of Vichy France, that was technically a neutral country, but Marshal Petan was subject to manipulation by the Germany.
Second, after Pearl Harbor, Germany declared war on the USA. FDR and Churchill agreed that UK-US forces would focus on defeating Germany first. The reason being that Germany dominated Europe and in 1941 had come very close to Moscow. Should the SU have lost European Russia, Germany could turn its attention back to Great Britain. Even though GB had survived Germany’s air attacks and won “The Battle of Britain,” Germany was a huge threat to GB with its U-Boats and another assault on GB. Should GB have fallen, the Allies would had a very hard time setting up the Western front.
Third, the North African campaign took some pressure off the Eastern front, as did the eventual Italian campaign, which put Italy out of the war.
_______________________________________________
Omar Bradley wrote in 1946 that FDR & Churchill did not even want to invade Italy, thought that they could let it wither on the vine. But, Overlord could not happen in 1943 and Stalin threatened a seperate peace with Hitler if we did not open a second (and in our mind, unnecessary) front.