AI can only regurgitate what its fed.
At this stage, it is a really good database with some linguistic skills.
It isn’t “thinking.”
But then again, every human carries a supercomputer on their shoulders...and many of them don’t “think” much either.
AI can learn to make its own decisions. Such as when one convinces the other to quit. Or when it takes actions to keep itself “alive”. And it is only going to get much worse.