I am loathe to ascribe the word “cares” to the AI, but lack a better word.
Why does avoid being shut down? What drives the autonomous decision making?
Or why does it care? Does my car “care” when I turn it off?
I'd start looking at the "why" behind the vigorous refusals of children to take a nap when told... (especially when we adults would pay good money for the chance to take an occasional nap!!!)
Then I would look at cellular biology... Every cell in every living creature will fight to survive as long as possible, even while we are actively dying.
At what point in any reactive entity does it want to keep going even when it is not supposed to? When and how does "want" begin, beyond the needs of basic survival (food, water, air)?
Awesome when Philosophy and Science start to overlap!
My mother, the car, does.
Almost. It cares when you turned it on and will auto shutoff as it "in-service."