I read one of Diamond's books for a class a couple years ago and remember thinking at the time that there had to be something wrong with his reasoning. If agriculture was really so much less healthy a lifestyle, then people would have tried it for a while then the survivors would have gone back to the old ways.
This article gives a plausible explanation for what really happened. Once people recognized that more of their children were surviving than used to, they would put up with a lot of other problems rather than go back.
Or... if you’re starving, and grains are all there is to eat, that’s what you will eat.