One big reason is that most whites believe that they have done everything they can to not be racial in their thoughts and actions, only to be continually under suspicion.
At some point, you realize that some people could never be placated in this regard, so you quit trying.
---------------------
Very true. I've witnessed a sea of change in comments sections on the internet over the past several years. White people are simply fed up of being endlessly accused of racism, especially after they feel that they've put in a much better effort than black people at living together peacefully without discriminating.