“If you’re a decent shot armed with a .45, the guy wearing body armor will be on his ass, head shot or not.”
If you were correct, the SHOOTER would also land on his rump. The bullet doesn’t have more energy at impact than it had leaving the gun.
It’s not the inertia that knocks you to the ground it’s the pain.
Don't do much shooting, do you?
The force exerted by the bullet on the target is, to first order, determined by the mass of the bullet multiplied by its acceleration, which would be negative as it slows down on impact. The force exerted on the shooter is also, to first order, determined by the same mass multiplied by its acceleration over the length of the barrel. The time part of the equation, and hence the value of the acceleration is much larger for the shooter than the target. So the force is less.
For typical body armor the deacceleration forces are quite high.
That is why when you watch a sniper using a .50 against a taliban perched on a hillside far away, the sniper recoil moves his shoulder a few inches, and the target actually cartwheels through the air end over end. Oh, wait.