Having to earn a living through hard labor is one of the punishments for sin. So is the estrangement and conflict between men and women, mentioned in the same place in Genesis. That doesn't mean God wants us to suffer these things at all times!
Gambling is a form of entertainment. If you can afford to spend $20 on dinner, there's nothing wrong with spending the money. If you can afford to spend $20 on gambling, there's nothing wrong with that form of entertainment.
Gambling is immoral only when one spends more time or money on it than he can afford, as with any other form of entertainment.