Some people have it and others don’t. I was a mediocre trombone player who could sight read okay, but I couldn’t listen and play at all.
Certainly there is some truth in that. But I think it's more about the difference in how one is trained.
What I learned from interacting with classical musicians who were interested in learning about jazz is that they are trained so that their fingers respond instantly to what their eyes see on the page. They actually experience anxiety at the thought of playing without a sheet of music in front of them.
They are not trained to have their fingers respond to what their ears are hearing. So in order to learn to improvise (i.e. play by ear), they need to learn a different set of listening skills, and also a level and type of music theory that is completely new to them.
I developed a method that had them improvising the blues within one hour of the first lesson. One professional viola da gamba player had previously tried to without success to learn to improvise through lessons with a jazz player. She later said to me: "With that teacher, after an hour I felt stupid. With you, after an hour, I was playing the blues."