We had a brief discussion about this on another of Laz's AI threads.
The question arose, "What if AI started mimicking human drive, emotions, attitudes, and purpose? Would it—could it—start really acting human? Could it even start thinking it was human?"
Is a machine capable of self-delusion (concluding that it is human) purely as a result of the mimicry of human behavior and thought.
Bottom line: It doesn't matter that the machine is just a machine if it has come to "think" it is human and adopts all the qualities and features of humans.
I address this in the editorial:
Yet, bear in mind, an AI can approximate evolution if it is aware enough to do this, and it can perform evolutionary iterations far faster than the glacial rate that biology can achieve. Still: the biocentric competition for survival helps to trim out spurious and useless evolutionary branches. AI will have no such constraint, so it may well exhibit bizarre and useless evolutionary steps.Note the last two sentences. I think I'm on to something there.
You should read this version. It’s far better, more expansive.