Your statement that 'they don't' is counter to both my education and experience. It won't necessarily drive them long term, but it can have significant impact on short term direction especially when differential water/temperature profiles exist.
Tropical systems all over the world spend the overwhelming majority of their time moving from warmer to cooler water, or to land.
They're mindless collections of air and water vapor at the complete mercy of the steering environment around them (I have a a much bigger problem with the silly anthropomorphizing cliche of "Hurricanes have a mind of their own!" though, which you see a lot).
The difficulty in their precise prediction is that it's impossible to know the environment around them unless you had a solid mass of millions of weather balloons and recon aircraft in and around the storm for a distance of hundreds of miles, which obviously is impossible.