I think every one must make their own decision. If you feel comfortable going then by all means go. If there is the slightest doubt perhaps it is your instinct warning you.
I used to go every two years or so, choosing a different starting point, such as Rome or Paris, but always ending in London. I was last there in 2014 and vowed never again and so far I have kept my vow.
The last time made me so sad witnessing the changes and the fading British culture. I was amazed at how many changes could happen in two years, and none of them good.
Best wishes for a safe and wonderful trip should you choose to go.
We went 10 years ago and had a good hotel in westminster Up the street near the tube station was large mosque. People always milling outside and taking over the sidewalk etc. We always walked on the other side of the street. My wife is a modest dresser and it was fall. We still got a lot of stares as if we were intruders.