Artificial Intelligence, Copywriting, and Creativity

The Dummy Guide to ‘Perplexity’ and ‘Burstiness’ in AI-generated content

Understanding Language Models: A Simplified Guide

AI is fast becoming a part of our everyday lives, and with that comes the need for general knowledge of its inner workings. So here’s a simplified explanation of two main concepts: perplexity and burstiness. Together, these two metrics can help us better understand how large language models write, and how we are able to detect AI-generated content.

Perplexity

Perplexity is a measure used to evaluate the performance of language models. It refers to how well the model is able to predict the next word in a sequence of words. As you’ll probably know by now, AI-generated text is procedurally generated; i.e. word-by-word. AI selects the next probable word in a sentence from a K-number of weighted options in the sample.

Perplexity is based on the concept of entropy, which is the amount of chaos or randomness in a system. So a lower perplexity score indicates that the language model is better at calculating the next word that is likely to occur in a given sequence, while a higher perplexity score indicates that the model is less accurate. Basically, the lower the perplexity, the more predictable it is. This indicates better generalization and performance.

As a really rough example, how do you think should this sentence end?

“I picked up the kids and dropped them off at…”

A language model with high perplexity might propose “icicle”, “pensive”, or “luminous” as answers. Those words don’t make sense; it’s word salad.

Somewhere in the middle might be “the President’s birthday party”. It’s highly unlikely but… I guess it might be plausible, on rare occasions?

But a language model with low perplexity might answer “school” or “the pool”. That’s an accurate, correct prediction of what likely comes next 🔮

As you can see, there are varying degrees of plausibility in the output.

(BTW, note how by accurately predicting language, AI gives the appearance — erroneously — of being factually accurate as well. Don’t fall for this fallacy:

Perplexity is commonly used in NLP tasks such as speech recognition, machine translation, and text generation, where the most predictable option is usually the correct answer. For writing generic content that’s intended to be standard or ordinary, lower perplexity is the safest bet.

Face it: most of the time, what we humans say and write is usually pretty boring! It’s easy to calculate what word comes… [next? after? tomato?]

Burstiness

Burstiness basically measures how predictable a piece of content is by the homogeneity of the length and structure of sentences throughout the text. In some ways, burstiness is to phrases what perplexity is to words.

Whereas perplexity is the randomness or complexity of the word usage, burstiness is the variance of the sentences: their lengths, structures, and tempos. Real people tend to write in bursts and lulls— we naturally switch things up and write long sentences or short ones; we might get interested in a topic and run on, propelled by our own verbal momentum. Like I did^

AI is more robotic: uniform and regular. It has a steady tempo, compared to our creative spontaneity. We humans get carried away and improvise; that’s what captures the reader’s attention and encourages them to keep reading.

How can I tell if a text was generated by an AI?

There are standard measures of burstiness and perplexity that are commonly used in natural language processing (NLP) and machine learning. To calculate these measures, you would need to use a natural language processing tool or library that can compute them. But human intuition will work in a pinch. Analyze the varying sentence structures, and count the number of unique words in a sentence divided by the total words.

You can finally put your college degree in literature to use! Judge writing. Is it interesting? Does it meander or stay on a topic too much? Are there any interesting words—or ones that seem out of place? These are all questions you can use to evaluate the perplexity and burstiness in a piece of writing.

There are also AI-based content analyzers like Originality.ai and GPTZero, if you’d like a more accurate assessment. It’s like the Space Race: pitching language model algorithms against each other in an escalating Cold War.

It’s important to remember that while AI-generated content can lack the variability of human content, that doesn’t mean that AI-generated text is completely devoid of entertainment. There are many examples of people producing unique, exciting content with AI. And if I’m perfectly honest, I find Ernest Hemmingway’s writing to be low in perplexity and burstiness!

“How can I improve my AI content so it appears more human-like?”

We’ll dive into that in a subsequent article, so make sure to click subscribe!

I’d like to emphasize that we’re not just talking about “How can I avoid my AI content from being detected?” There’s no cheat or easy fix; it’s going to require some effort on your part. The same techniques that will make your text sound more human will also vastly improve the quality of your writing.

However, I will say that in the case of AI content detection, if the text is predictable by an AI model, you will trigger a detection algorithm. Basically, GPTZero and Originality ask: “Could I have written this?”

A lower perplexity score indicates that the text is likely machine-generated. So if you wanted to avoid detection, it may be beneficial to generate text with a perplexity score that is closer to that of human-generated text.

This can be achieved by using more advanced language models that have been trained on larger datasets and have a higher level of complexity. For example, Jasper AI implements a collection of large-scale language models (unlike ChatGPT). This means it incorporates more syntactic and semantic features to generate text that is more complex and challenging to predict.

Currently, Jasper is undetectable by GPTZero and Originality, and others.

As for burstiness: edit for diverse and complex language patterns. It can be a good thing if the bursts are contextual and pertain to a particular topic. If the burstiness is random, it may not be as useful. Generally, it’s important to aim for a balance between low burstiness and high coherence to create natural and engaging content that is less likely to be flagged as spammy.

Don’t let AI take the reins of your writing — you’re in charge.

But remember, if you want your content to stand out and capture readers’ attention, you’ve got to flex your creative muscles. The best AI content is more truly cyborg content: your creativity and inspiration combined with the productivity and proficiency of new technology. You’ll still always need to find the right combination of magic words that will resonate with your audience and make them interested in what you have to communicate.

Who is Jim the AI Whisperer?

As The Jasper Whisperer, I provide training and consulting services to help companies prepare for and properly utilize AI in their operations. Don’t miss out on the huge benefits of AI for your business. Take control of the technology and make informed decisions. Contact me to learn more.

I’m also available for journalism opportunities, podcasts, and interviews.

Ready to join Medium?

Gain unlimited access to the entire Medium catalog with my referral link, and you’ll also be supporting my ongoing writing at no extra cost to you:

You might enjoy these related articles from Jim the AI Whisperer: