The version I always heard was that the rock taken out weighed more than the skyscrapers that were built.
As for buildings that aren’t skyscrapers built on land that isn’t bedrock, is sinking usually that much of a problem?
Manhattan is a very interesting study in geology. Lower Manhattan is all rock. Therefore, the large buildings especially down by Wall Street.
Then going north there is very little of any height until you get up to the Empire State building on 23rd street. Then a whole bunch up through Mid Town and on each side of Central Park.
Greenwich Village is all land filled swamp.