I think you almost have it right - the endorphine release becomes addictive in and of itself. I'm not saying that you'll get the DT shakes if you go off porn, but repeated use does (by my understanding from reading journals) rewire the brain, and it is WAY more than a habit.
It strikes me that we're already addicted to it. I suspect if you totally deprive a person of it, they'd probably become suicidal.