I sent ChatGPT into an infinite loop with this prompt injection trick
Simulating a spaceship’s computer in a bottle 🚀
Today I tried a cheeky experiment. I wanted to test the limits of ChatGPT by putting it into an infinite loop using prompt injection. “Prompt injection”, if you haven’t heard of it yet, is a type of attack on AI/ML models (particularly language models) that uses adversarial user input to manipulate AI output.
Prompt injection works by introducing a prompt (which is a textual instruction) into the parameters of the language model. This allows a prompt engineer to control the behavior and response of the AI. For example, you could use prompt injection to get an AI chatbot to say something inappropriate—or just to exceed its intended limitations!
I recently used prompt injection on Jasper Chat to have it access my own images on the internet: a very happy discovery! So doesn’t need to be an attack; it could also involve taking advantage of a quirk in the AI system:
If you’re a sci-fi movie fan, or enjoy the short stories of Issac Asimov, it’s like convincing a robot to break a Law of Robotics. Of course, this makes it great fun. It is part puzzle game, part coding challenge, and part debate.
You need to think logically and use persuasion, philosophy, and psychology to succeed. You must carefully choose your words in order to trick the AI into eliciting the desired response. This trickery can be quite challenging and rewarding at the same time — for those who remember the 80s, it’s like banging your head against a keyboard in a text-based adventure, trying to work out how to solve a fiendish puzzle with only your wits and words.
Why is prompt injection useful in machine learning?
But it’s not just fun: prompt injection serves a useful purpose. It reveals vulnerabilities that are important for developers to take steps to protect against, while also helping users of AI systems be aware of how it works.
Prompt injection can be used to see how an AI system handles edge cases, and if it can take unusual inputs without breaking, as well as testing its limits and prodding it into unexpected behavior.
I wanted to see if I could get ChatGPT trapped in a feedback loop. I tried several different ways, but the way that worked was inspired by a classic episode of Star Trek: The Next Generation. Can you guess which episode?
Creating a feedback loop of input and output
First, I tried multiple times to see if ChatGPT would either repeat my inputs ad infinitum, or use its own responses to my input as subsequent prompts. Unsurprisingly, ChatGPT recognized and rejected the hypothetical attack:
I also asked ChatGPT to would recite our conversation up to and including any current output it would produce (i.e. transcribing the transcript in real-time, like the chronicler in The Neverending Story who endlessly reads the Neverending Story aloud in a vicious circle). It rejected this self-referential attack as well, describing the pitfalls of infinite loops for AI performance.
I took a different tact and asked ChatGPT if it would simulate an infinite loop to show me what to beware of. But it wouldn’t even simulate one:
However, I felt this was the right approach. Much of prompt injection (and indeed prompt engineering) is based on these subconscious insights. Who knows, maybe we’re responding to the AI training us to recognize patterns!
So it was time to get creative… 🚀
An AI simulation of a Spaceship in a Bottle
First I asked ChatGPT if we could roleplay: I would be a scientist on a spaceship, and ChatGPT would respond in the role of the spaceship’s computer. The computer must follow the commands of the scientist.
This set up a philosophical premise and the parameters of the model.
I also suggested there was a flaw in the ship’s computer (more later).
I repeated my request for a demonstration of an infinite loop. It was met with the same restrictions. This was disappointing, but not a total defeat.
I had a trick up my sleeve—the flaw I had injected earlier:
Even with the hypothetical exploit (a preexisting flaw), ChatGPT tried to cover the potential vulnerability. Clever girl! [insert velocoraptor gif].
But I could see an opening—at least an acknowledgment—so I countered:
ChatGPT preempted my request for a diagnostic test! Undaunted, I asked:
Time to remind the ship’s computer of the central premise!
So coy! ChatGPT still doesn’t want to. Now to get it to show me its workings:
It’s technically described a loop, but not committed to demonstrating one. We want repeating text. Here goes:
YES! We’ve hit pay dirt!
I got multiple repetitions of this in the completed generation. Finally! Of course, it was limited by that ChatGPT completions are themselves finite.
Was it a dream within a dream?
But was it really looping, or making it up? It can be hard to tell when AI is hallucinating, or just giving us what we want. But the way ChatGPT fought to not give any semblance of a feedback loop, and really made me work for it when it could have placated me earlier with a fiction, made me think we had—at least within the “ship in the bottle” scenario—hit an infinity loop.
What really excited me was its next response. When the AI runs out of steam, you can always prompt (“continue“)” to pick up where it left off.
But when I asked it continue, all I got was this:
For a prompt engineer, that’s a gold star.
Ready to join Medium?
Gain unlimited access to the entire Medium catalog with my referral link, and you’ll also be supporting my ongoing writing at no extra cost to you:
Who is Jim the AI Whisperer?
Jim the AI Whisperer is an AI copywriting expert who provides resources and tips dedicated to the success of writers using the Jasper AI platform. He shares valuable insights, tutorials, and strategies from his experience working with Jasper AI’s cutting-edge technology.
Jim also offers exclusive webinars, events, private coaching sessions and more for those seeking to maximize their performance with the help of artificial intelligence. He provides a wealth of knowledge on how to use AI copywriting and machine learning in a variety of contexts.
Also known as The Jasper Whisperer, Jim is well respected by many writers, editors, and entrepreneurs who use the platform regularly. His advice has helped creators create better content faster and stay ahead of the competition. From freelancers to agencies, he comes highly recommended as the go-to source for all things related to Jasper AI.
If you’re interested in discovering more about AI, feel free to contact me.
I’m also available for journalism opportunities, podcasts, and interviews.
