I have the "annoying" part in quotes, because I wouldn't mind hearing sonic booms at all.
When I was a little kid, I used to hear them back in the late Fifties/early sixties, from military aircraft, and when I was in the Navy, they used to put on excellent airshows that included supersonic flight deck level flybys that we were able to watch from the flight deck.
There was something uniquely alien, bizarre, and cool about seeing an F-14 Tomcat, wings swept back approaching at flight deck level at supersonic speed.
When it approached, you could see flashes of compression that appeared and disappeared in the blink of an eye. The speed that they approached at was, for me, speed I had never seen in anything that gave it an alien appearance...not of this Earth.
What looks like mist in the image above would appear and disappear in what I guess would be a tenth of a second...if you blinked, you would miss it! Then in a flash, the plane would sweep by in front of you without a sound. Which was creepy and amazingly fantastic and cool at the same time. Zero sound. As I said, it appeared alien and unearthly.
Then, about a second later, depending on the speed of the plane...WHAM!
If you were looking the wrong way and weren't expecting it, is sure made you jump, even if you had heard it before!