Think you’re getting away with AI-generated text?

Think again.

Computer science student Edward Tian built GPTZero, an application to check if a given text is generated by AI. It looks at PERPLEXITY (evaluated on GPT2) and BURSTINESS.

The GPTZero website explains it as follows:

PERPLEXITY — i.e. the randomness of the text is — a measurement of how well a language model like ChatGPT can predict a sample text. Simply put, it measures how much the computer model likes the text. Texts with lower perplexities are more likely to be generated by language models.

Human written language exhibits properties of BURSTINESS: non-common items appear in random clusters that will certainly appear over time. Recent research has extended this property to natural language processing, some human written sentences can have low perplexities, but there are bound to be spikes in perplexity as the human continues writing. Contrastingly, perplexity is uniformly distributed and constantly low for machine-generated texts.

I tried it out eight times with different texts, and it gave an accurate verdict each time. The site is incredibly slow, often gives a time-out error, the application is probably far from perfect, and my test was by no means thorough. I’m curious to learn what AI scientists think of it.

In terms of the societal impact of generative AI, it marks the start of a new technological cat-and-mouse game. AI gets better at generating human-like written text; tools get better at assessing if a human or AI wrote it. Quite like what’s going on in the (fake) news industry. We have some interesting times ahead!

You can check it out at gptzero.me

If you’re interested in the impact of new technologies on industry and society, follow me to stay in the know.