You're kidding yourself. Buoyancy is a simple scalar property. If anything, you lose buoyancy as you sink due to compression. There is no tendency to become more buoyant with depth.
I myself am near enough zero buoyancy that I can float, then expel air and sink. This is how I do my "lobster imitation".
And at what point do you stop sinking if you’re holding your breath? Are you on the surface or are you a few feet below? I taught kids to swim for years. If my meaning was how you’re erroneously interpreting, no one would ever be able to get to the bottom of the deep end without swimming there, and would have a difficult time remaining without popping back up like a cork. That is obviously untrue.