I had three employers over my career, and I did indeed keep my 401K's together... simpler to manage that way.
I've always looked at it as "1. 401(k) investments are generally worse, higher cost and fewer choices than I could get on my own at a low cost broker (Vanguard, Fidelity, Schwab, etc.). 2. I would rather have control of my money rather than let someone else control it, especially if things start going bad at my current job." As soon as I leave, the money gets rolled over into my personal account. The only thing I lose is the ability to borrow against my 401(k), but I have never been tempted to do that and my last job didn't allow it anyway.