IIRC, the uniforms for all branches of the German military (wehrmacht, luftwaffe, and kriegsmarine) did *not* have the swastika armband (which I think was reserved for civilian party rallies and SA thugs).
If this is the case, then Harry was needlessly embellishing his already stupid choice of a costume.
You're right. The regular armed services did not wear it. It was only the real thugs you mentioned, including the SS, who wore it.