Empire State Building was seen as a folly being built in the middle of the depression. They flooded the construction site with guys willing to work cheap. They had lines of guys around the block looking for work. If you got hurt up on the steel, they'd just send down for a replacement who would work just as cheap.
That's certainly a part of it, but not all of it. We can't get plans off the table in 15 months these days, let alone build the tallest building in the world in that time, or twice that time.
I think 15 or so men died in the building of the ESB. I suppose that's sexist - 15 women should have died, too.