That's what I was thinking. It looks like he's pointing to the Sixth Avenue area if you estimate the distance to Penn Station.
I used the Empire State Building on 34th and 5th as the point of reference and moved from there.