Summary

The article discusses the limitations of ChatGPT in generating code for applications, emphasizing its unreliability for complex tasks and the necessity for user intervention.

Abstract

While ChatGPT can impressively handle simple coding tasks, such as creating a palindrome checker, it struggles with more intricate applications, like a tic-tac-toe game. The limitations of GPT-3.x, including token size restrictions, outdated knowledge, and lack of optimization, hinder its ability to produce fully functional applications in one go. Users must be specific in their prompts, ready to update deprecated code, and capable of optimizing the generated code themselves. Despite these challenges, ChatGPT can still serve as a valuable educational tool and coding assistant when combined with software engineering best practices.

Opinions

ChatGPT's code generation is often suboptimal and may not execute properly without user intervention.
For educational purposes, ChatGPT is considered a reliable teaching assistant, particularly for basic coding examples.
The article suggests that users need to have a clear understanding of what they want and how to ask for it to effectively use ChatGPT.
The limitations of ChatGPT, such as its knowledge cutoff and inability to produce optimized code, necessitate user involvement to adapt and refine the output.
Despite its limitations, ChatGPT can be a powerful tool when used in conjunction with software engineering best practices.
The article implies that users should not expect ChatGPT to replace manual coding efforts, especially for complex applications.

Coding with ChatGPT, 5 limitations to keep in mind

In this article we’ll look at ChatGPT’s limitations and how they impact its ability to generate full-fledged applications in one shot!

We’ve all seen online videos of people excited, or terrified, by ChatGPT’s ability to generate code. But very few took the time to read the code and execute it. And that’s the catch… Most of the generated code is at best suboptimal (for basic tasks); more often than not, the code won’t pass the execution phase.

Simple toy example, is this text a palindrome?

Let’s be fair, for well known toy examples, ChatGPT will do great! The main requirements for it to work are:

Implement a well defined function.
The code fits in a few lines.

ChatGPT coding a palindrome function with a few test cases in one output.

Does it work if I copy past it in Colab for instance?

ChatGPT coding a palindrome function with a few test cases in one output — Colab Version.

Yes! It does with no changes from my side! For these simple cases, mostly educational, ChatGPT proves to be a very reliable teaching assistant!

ChatGPT can be a very powerful assistant provided you know what you want and you know how to ask for it

Still a simple example: build a tic tac toe HTML, CSS & Javascript application

It is still a toy example. Yet, it proves to be challenging for ChatGPT.

Let’s look at this example: in the video below, Conner Ardman asks ChatGPT for a simple tic tac toe application with HTML, CSS and Javascript (so very basic stuff), and yet, ChatGPT has a hard time generating concise and exploitable code. The video fails however to explain why we struggle so much to build something efficient relying on GPT3.x in general.

So… why is it that ChatGPT struggles with such simple tasks? In this article we’ll look at ChatGPT’s limitations and how they impact its ability to generate full-fledged applications in one shot!

GPT3.x Limitations that hinder its ability to build applications

If you want to learn more about GPT3.x shortcomings (and why it won’t replace Google anytime soon), please refer to this article: https://readmedium.com/why-chatgpt-isnt-ready-to-kill-google-yet-b2672e464f82

When it come to coding, here are (at least) 5 limitations to keep in mind:

Input/Output size limitation: GPT3.x is limited to 4k tokens for the sum of input prompt and the outputted result. I’m not sure of the actual size limitations for ChatGPT, but it is safe to say that you can’t expect it to return a full application if its content is too long. As a result you might see the model stop in the middle of its code generation (as it happens often in the video).
Limited Recent Knowledge — Obsolete versions: Langage models rely on the massive knowledge encoded in their neurones. Their main task is text completion ; thus when asked to build an app, they’ll naturally rely on most seen similar code, which most likely old. As a consequence, it will often rely on older versions, library definitions and features. If you need recent version you’ll need to ask for it specifically in the prompt (assuming it’s available in its knowledge bases. Remember that GPT3.x’s knowledge is limited to June 2021. Any library or framework update that happened after that date, is unknown to the language model!
Limited Recent Knowledge —Deprecated endpoints : If you are building an application that relies on APIs endpoints, it’s very likely that you’ll get deprecated endpoints. Same reasons as the previous point, i.e., the langage model has no knowledge beyond a certain cutoff date. You’ll have to tweak the code to adapt the used endpoints definitions.
Optimization Issues: Language models won’t natively provide you with optimized code ; they’re not supposed to code in the first place! It is still a text completion model, and code … is text. If optimization is of importance to the task at hand, you’ll have to either update the generated code yourself or ask the model to optimize the code in a certain way. In all cases, you’ll need to know what you are looking for.
Hallucination: if the prompt isn’t specific enough, the model will make shortcuts and return a code with undefined functions supposed to solve parts of the problem (or that shouldn’t be there in the first place). You’ll have to ask for details about those functions or implement them yourself when it makes sense.

As a result, if you’re hopping that ChatGPT will generate a full application for you, think twice, because you might end-up spending more time debugging it than actually coding it by yourself!

Closing Words

All of these issues can be mitigated, provided we rely on software engineering best practices (e.g., modular files and architecture).

As a matter of fact, ChatGPT can be a very powerful assistant provided you know what you want and you know how to ask for it. But it won’t generate a full feldged application from the get go.

In the next post, we’ll rely on ChatGPT, and software engineering best practices, to build a python application. Stay tuned ;)!

If you liked this post, please consider supporting us: 🔔 clap & follow 🔔