“Interstate” tolls which the Federal government has to agree to.
I’ll defer to locals but afaik if you’re south of the Park - or indeed anywhere south of the north end of Central Park - you can take the Lincoln or Holland tunnel to get to NJ. The only other option to avoid the toll is to drive way up to Washington Heights to take the bridge but that drops you off way north (or vice versa if you’re coming from NJ to NY) and probably adds a lot of commute time on both sides of the Hudson. So yeah seems an unfair tax on commuters from the other states.