If even people who didn’t vote for Trump and wouldn’t vote for him next time are telling pollsters they thought the election wasn’t wholly honest, they probably aren’t up in arms about it. The world is funny that way.
For the people who "wouldn't vote for Trump", what difference does it make if they don't "like hearing Trump (or anybody) go on about it"?