“It’s when the people who can afford say “nope” that’s when prices drop.”
But you’ve answered your own question. The people who can afford to stay are not leaving, so the average income level is rising. The home across the road from us just sold a year ago to a young couple “moving up.” They have grade school kids, and enough income to purchase a $2.2 million home! Farm workers aren’t going to ever afford that kind of home price, and as agriculture automates, they will be forced to leave.
The other issue respecting SF Bay Area housing is that there isn’t any land left. Housing isn’t affordable because there is no stock, and little prospect of any large-scale building of same. As it is, people are being forced to commute to their jobs from the Central Valley. I had a guy working for me in a San Jose high-tech company as a warehouse supervisor. He was driving 50,000 miles a year just commuting!
Most of the buildable land is gone. South San Francisco wanted to build a high-rise residential complex in my neighborhood, 15 stories or more. My neighbors screamed at the city council and got them to reduce the scope to 8 stories. Our neighborhood is mostly 1 and 2 story residential homes. City leaders say we need the housing, and it's being built a couple blocks from the BART station, so they just approved a builder. Only way to create more housing is to go up, high-rises. Jobs are plentiful, housing is scarce. Even after building these complexes, it won't keep up with demand. More tech companies are building here and bringing in more workers.