[PJBankard]

That’s still doing what it is programmed to do. It isn’t doing anything that it wasn’t programmed to do. An unforeseen outcome is not AI if the program did exactly what it was programmed to do.

[RoosterRedux]

Algorithms can be written in which an agent or program can encounter something new--a problem perhaps--and learn how to solve it and adapt to it and then incorporate that change into its code. And then when the next new problem is encountered, it has a newly acquired or adapted method for seeking a solution that was not programmed into it.

This continues until the end program is nothing like the original algorithm.

I am guessing the reason for shutting down this pair of agents was not because of fear of independent thinking but because the scientists were losing the ability to monitor the changes made to the algorithm due to the development of a private language between the two agents.

Doesn't make sense to create a self-developing algorithm if the algorithm refuses to let you see what it has developed.;-)