I know several people who were in your exact position. Then 2009 happened.
The areas I have housing in only lost 10% in 2008-2009 bc they are a lot more steady and rents went up. I’m also extremely diversified and will reallocate to cash and bonds if I think a recession is imminent. I got out of stocks in early 2008 last time although I waited until 2012 to get back in significantly so missed some upside. Plus I make a lot from day job and am saving 100k/yr. I’ll be fine.