Apples and oranges again. 20,000 of approx. 18 million is a tiny fraction of the overall troop strength. Whereas Muslims constituted a plurality of the FFL's African troops, a group that was itself a significant portion of the FFL.
The only fair comparisons would be to present either both as raw numbers or both as a percentage. When compared in the latter sense, a couple thousand muslim africans on the war's periphery is a virtually negligible part of the 100-something million or so participants in the allied side of the war effort. IOW, those 20,000 jihadi SS troops constitute a greater percentage of Hitler's forces than the colonial muslims did of the Allied forces.