The minorities. There would be 50 witnesses saying they were the attackers.
In most cases, it is common sense. A few years back, a couple of black kids decided to lay it into a larger crowd of white kids on a train platform. The white kids fought back. No hate crime charge, and all witnesses present verified that the others started it.
Don't mean to get too racial here, but this is a racialized discussion anyway.