Your concern is well placed. There are at least two lawsuits I know of regarding teenagers who used AI chatbots who later killed themselves- one with the clear encouragement of the AI
That’s horrifying.
Some will say “well those kids were close to the edge anyway, i would imagine”
Even if they were, does that mean we have to add things to the last that would push them over?
They are not ready for the complex emotions of being in love with a HUMAN yet...no less what the heck this thing is.