I think Hitler’s and the Nazis’ extreme enmity toward Communists (the Left) contributed to their being labeled on the Right; i.e., they must be the opposite of the Left. A bit simplistic, perhaps, but it makes sense to me.
Yeah but we are outside both those camps. It’s sort of like looking at Sunni and Shiite Muslims. Those groups have extreme enmity towards each other too but from the outside they are just varying flavours of Islam.