Sort of correct. A receiver can’t change direction to occupy space already occupied by the defender (or about to be occupied due to momentum) and it be interference. The defender has a right to a spot on the field. It’s the same as if a middle linebacker is standing in one spot, the ball is thrown directly toward him and then the intended receiver runs into him. Patriot whiners would have us believe that is pass interference.
More often than not, it is called pass interference on the defense. The officials need to crack down on this, as it's becoming a tactic to intentionally underthrow a receiver who comes back to the ball through the defender, and picks up a pass interference call for a cheap big gain as the defender can't change direction fast enough to avoid the collision.