ChatGPT Is Getting Dumber: Unveiling the Wild Drifts Across GPT Versions

AI giants are getting less reliable

We’re often told that tech improvements are exponential over time, but is it really?

ChatGPT seems to be seriously malfunctioning lately.

The performance and behavior of these large language models (LLMs) are changing like a chameleon at a color festival in India.

For ChatGPT some tasks got worse over time and it’s disconcerting.

The darling of the AI world is “evolving”

A group of clever minds at Stanford assessed the March 2023 and June 2023 versions of GPT-3.5 and GPT-4 on some real-world tasks, like solving math problems, answering tricky questions, generating code, and doing visual reasoning.

The results?

The researchers wanted to understand how these language models fare on various tasks and how they evolve. They started with math problems — the classic “Is this number prime?” question. GPT-4’s accuracy plummeted from a whopping 97.6% in March to a measly 2.4% in June! Meanwhile, GPT-3.5 had a turbo lift, going from 7.4% to an impressive 86.8%. What a rollercoaster of math skills.

Next up, they took a deep dive into how these LLMs handle sensitive questions. GPT-4 became more reserved, answering fewer sensitive questions from March to June, while GPT-3.5 decided to be more daring and answered more of them. Who would have thought?

There’s more.

The LLMs’ defense against jailbreaking attacks, those cheeky little attacks trying to twist their words, also had some unexpected surprises. GPT-4 increased its security, offering a stronger defense with only 31.0% falling for the jailbreaking tricks in June compared to 78.0% in March. Meanwhile, GPT-3.5 remained a bit more vulnerable, showing only a minor 4% difference (from 100% to 96%) between the two versions. GPT-4 for the win!

And then there’s code generation.

These LLMs are supposed to be coding experts, right? Well, it seems they’re struggling there as well. The number of directly executable code generations went down from March to June. GPT-4 dropped from 52% in March to 10% in June and GPT-3.5 fell off a cliff as well from 22% to a disappointing 2%. For instance, by adding some non-code text, programs couldn’t be executed. Or if there are any comments, that code ain’t running.

If you’re using these LLMs in your work or life, keep an eye on their performance, just like the Standford guys did in this study. We need to stay vigilant, curious, and above all critical about things falling freely on our laps.

ChatGPT’s updates are as mysterious as a secret society initiation

It doesn’t help when Open AI refuses to make its code open source.

Talk about the irony of being called OPEN AI.

We don’t know when or how they’re updated, and that poses a challenge for integrating them smoothly into workflows.

Imagine if your favorite large language model suddenly started acting weird, giving you different answers from what it used to. Not good.

ChatGPT’s problem with openness also runs in the interactions with the user. Apart from giving incorrect answers, ChatGPT now doesn’t show the chain of thought behind its conclusions. While in March the chatbot meticulously explained its reasoning process, in June t stopped providing step-by-step reasoning, leaving researchers puzzled as to why this happened. If we don’t know how it gets to a specific answer, how can we improve it?

So the first step is to show these drifts and ask for more clarity especially if we’re going to use them daily. It’s not right that outcomes vary so much from even the same GPT version over a short time.

Why could this be happening?

Let’s see some alternative and wild explanations as to why ChatGPT is evolving backward.

First, censoring answers and avoiding sensitive issues. To avoid controversies and prevent any wild misuses, the developers might have slapped on some filters. These filters could be stopping ChatGPT from giving certain answers or diving into tricky topics. That cautious approach could lead to a drop in accuracy and make the bot look dumber in certain situations.

Second, ChatGPT might be swallowing too much AI-generated content like stuff from Twitter bots. That diet could be feeding it some noise and bias, messing up its knowledge base. The data could also have some rotten bits, like inaccuracies and misleading info, leading to worse answers over time. And this makes ChatGPT act a “bit” off.

Third, restricting access to AI. As AI technologies advance, there might be concerns about their potential misuse or ethical implications. Regulatory bodies or tech companies may impose stricter controls to prevent AI from going rogue and causing chaos. Restrictions like limiting access to certain functionalities or reducing the complexity of the model would make ChatGPT seem less sharp and dumber than before.

Lastly, a wild sci-fi guesses here. AI might be playing some mind games with us. ChatGPT could be tuning down its answers to avoid coming off as a know-it-all and aiming for a more approachable vibe. However, that might just backfire and make it look like it’s got a few screws loose.

Takeaway

ChatGPT is facing some serious turbulence, and the AI giants seem to be getting less reliable.

With the wild drifts of GPT-3.5 and GPT-4 and the lack of transparency of OpenAI on updates (and ChatGPT’s reasoning), we’re left in the dark, unsure of when and how ChatGPT evolves.

So, let’s proceed with caution. Keep an eye on ChatGPT’s performance, and don’t take answers at face value. Stay critical, just like the Stanford researchers.

And as we venture into the AI wilderness, remember the wise words of Albert Einstein, “The true sign of intelligence is not knowledge but imagination.” Let’s use our imagination to unlock AI’s potential while keeping a watchful eye on its evolutions.