I find that very hard to believe. NYers LOVED the Towers. Loved them. Especially after 1993. Maybe it is you who needs to get out more.
New Jersey people (like me) loved the World Trade Center, not New Yorkers.
The fact that the Port Authority, the bi-state agency, owned them and that the easiest way to commute to the WTC was from New Jersey and that you didn't have to go outside once you got there all made the complex very appealing to NJ people. Also the best views of the WTC were from NJ.
The twin towers were our piece of New Jersey in New York.