Here's why:
This is a shadowgraph of a bullet in supersonic flight. It shows the shockwave (the cause of the supersonic crack) propagating outward from the bullet in a conical shape.
Because the shockwave is sound, it travels at ... guess what? ... the speed of sound. Because the bullet is traveling faster than the speed of sound (and you can tell from this image that this particular bullet was flying at more than the speed of sound because the angle of the shock cone is less than 90°), relative to the bullet's trajectory, except for that tiny bit where it's surrounding the bullet, the shockwave is entirely behind it.
That means it's impossible to hear the bullet until after it's gone past you. So if you were standing where the red oval is (not drawn to scale) and facing directly toward the shooter, you couldn't yet hear the bullet's crack even though it already has gone past you. And you won't be able to hear it until it has traveled far enough downrange for it's shock cone to pass over you.
Which means, as I've drawn this, when you finally do hear the crack of the bullet, it will sound as is it's coming from behind you and over your left shoulder. And presuming that the shooter was at least a good enough of a shot to get somewhat close, your sense of hearing will tell you that the shot came from behind you.
Watch videos of hunting feral hogs at night with NVGs and suppressed rifles and you can see this in effect. Whatever hogs are still alive will tend to ruin from the direction they heard the 'crack' coming from, which sometimes puts them running in the general direction of the hunters.
The silencer doesn't make the hunter silent, but it does make him (or at least his location) invisible.