Does it matter? Antifa wanted violence, they got it.
Yes, labels matter because the more there are the more confusing knowing who the enemy is. The whole of it has been contorted by splinter factions and the media preys upon that.
White Supremacist is not a label good conservatives should desire.