That’s nice in theory, but in practice doesn’t work so well.
Going back in history just a little, there were plenty of innocent Germans and Japanese whose innocence mattered little when it came down to it.
And they didn't round up and murder those innocent Germans and Japanese (as well as Italians) and kill them (which is what some people seem to be advocating here) and in retrospect, a lot of people don't think that what was done to them was unjust. You may also notice that the Nisei soldiers, Japanese who fought for the United States during WW2, were commended for their brave service.
Yes, people ignore innocence when they are frightened but it doesn't mean that what they do when they are frightened is moral or just.