People look at some of the ruins and say, "but where did they go?"
One year of bad harvest? Not a good time but survivable.
Two? You are starting to lose people.
Three? You have now lost people and animals you can not afford to lose.
Four? Your chances of having enough healthy people left to do any planting is minimal.
Five? Dead.
Egypt had Seven............
There's actually an undercurrent on YouTube, videos of people trying out the local English weeds, which were availed of back at that time, and probably before and after, because that's how people used to live before our current era of plenty.
My great-g's pioneered here. By contrast, if I'd been a pioneer, I'd have been dead in about five minutes, either via accidents (tree falling on me when I'm trying to cut it down, for example), starvation, or being eaten by some local wildlife.