ChatGPT: Student builds app to sniff out AI-written essays

bbc ^

[BenLurkin]

The app works by looking at two variables in a text - perplexity and burstiness - and it assigns each of those variables a score.

First, the app measures how familiar it is with the text presented given what it has seen during training. The less familiar it is then the higher the text's perplexity is, meaning "it's more likely to be human-written", Mr Tian said.

It then measures burstiness by scanning the text to see how variable it is. For example, does the text have a mix of short versus long sentences? Or does the writing appear to be more levelled and uniform?

"If you plot precisely over time, a human-written article will vary a lot," Mr Tian said. "It would go up and down, it would have sudden spikes."

He is still working on improving GPTZero, but he has released a beta version for public use. In a tweet, he demonstrated how the app can successfully sniff out the difference between an essay published in the New Yorker magazine versus a letter written by ChatGPT.

He said he has also since tested it out by feeding the app BBC articles written by journalists, versus articles written by ChatGPT using the same headline as a prompt. (Mr Tian formerly worked with the BBC's investigations unit). He said the app successfully guessed the difference between the texts with a less than 2% false positive rate.

Since its launch, Mr Tian's app has been used by thousands of people. He said he has since been contacted by teachers and university admissions officers from around the world who are interested in how it works.

[Raycpa]

I tested it by asking it to write a sermon on some verses about God’s authority I plan on using this Sunday.

I don’t know how to say this exactly but the result was well written but one dimensional. It fails on making the scripture relevant to our personal lives, fails on connecting other attributes of God in context and fails on connecting to other relevant scripture.

So, in short I still have to write my own.

[The Duke]

"I've seen things you people wouldn't believe. Attack ships on fire off the shoulder of Orion. I watched C-beams glitter in the dark near the Tannhäuser Gate."

[VetoBill]

So now we have the potential for software spying on other software?
We’ve always been afraid of AI taking over the world; we never considered a civil war among AI.