A Follow-Up Question for AI Test Automation Fans: Did you actually implement Real UI Test Automation with ChapGPT or like, before ChatGPT’s first anniversary?
Prove it or silently admit it is just another hype. Testers verify.
The biggest IT news in recent days is about OpenAI, the company that created ChatGPT.
This reminds me that the ChapGPT will celebrate its first anniversary next week. ChatGPT first public release: 2022–11–30.
There has been a lot of hype about ChatGPT for test automation. For example, there are even videos that “teach you how”.
As a tester, it is fair to ask, “Did anyone get it to work? I mean, for real”. By the way, I refer to the objective criteria, such as the Definition of End-to-End Test Automation Success.
I highly doubt it. To my knowledge, every so-called AI Test automation attempt failed, if put into real test.
This raises some interesting points:
- ChatGPT took the world by storm for its simple usage
Unless you have lived under a rock for the last year, you must have heard and used ChatGPT.
ChatGPT currently has 100+ million users, and the website sees nearly 1.5 billion visitors per month. — source
It is simple to use ChatGPT. Many school students are using ChatGPT (mostly for writing). Surely, it is not a skill that requires several months of training.
Whether ChatGPT will be useful for work reflects one person’s understanding of the work.
2. ChatGPT-related activities are welcomed by executives.
IT executives generally will prove AI and/or machine learning initiatives (cool). Even Microsoft Bing and Office offer ChatGPT integration. My daughter is still studying at university (part-time), and via her, I know that many of the IT research topics are AI/Machine-Learning related. This leads me to think, “What did Computer science researchers do previously then?”
3. ChatGPT is supposed to make Test Automation “easier”
If someone claims “ChatGPT revolutionize test automation”, I would assume they mean ChatGPT will make it easier, i.e. creating automated test scripts quicker and easier to maintain them.
I have seen some demonstration videos, but as an experienced automated tester, I don’t get it. I found it a complete waste of time (not helpful to the challenging work in real UI Test automation) using ChatGPT or other AI in E2E Test automation, at least for now and in the foreseeable future.
4. ChatGPT v4 claimed several hundred times more powerful
So, if someone thought ChapGPT was a bit helpful with UI Test Automation 11 months ago, surely the results would be a lot better with the new version of ChapGPT, right? To use ChapGPT v4, only $20/month, that shall not be a problem for AI Test automation fans.
One year is a very long time to prove an End-to-End Test automation approach (web test automation, the most common form of software testing, has not changed at all). I usually had it down on the first day. Check out my article, “How do I Start Test Automation on Day 1 of Onboarding a Software Project?”, and also my daughter's “Set up, Develop Automated UI tests and Run them in a CT server on your First day at work”.
In summary, for AI Test Automation believers, there are really no excuses for not delivering solid results. If not, there is only one explanation: these people don’t understand End-to-End Test Automation at all; they are fake.
For IT Executives or Start-up owners: finding an objective E2E Test Automation Approach is simple and quick. Check out Test Automation and Continuous Testing Competition Week.
AI Test automation fans, I welcome you providing evidence to prove me wrong. Here is a benchmark you can use to compare against.
This is just one of several E2E Test automation implementations I have been maintaining for years in my spare time. Yes, I consider myself quite productive at developing/debugging a large number of raw Selenium RSpec tests in TestWise and managing executions of them in the BuildWise CT server. I just don’t see ChapGPT or other AI tools that can match even 1% of my productivity level when it comes to real web test automation (maintaining 200+ comprehensive End-to-End UI tests that run daily as regression testing).
This suite (for the WhenWise app, as I develop in a true agile spirit, the app changes rather frequently) consists of 570 user-story-level raw Selenium WebDriver + RSpec tests. Below is a yesterday's run.
After getting the GREEN (passing all tests), I deploy the build to production. I have always been doing daily production releases. Yes, I do trust my automated E2E tests.
Related reading:
- “Actions beat arguments” in End-to-End (via UI) Test Automation
- Why Training ChatGPT Won’t Get Good Results in Test Automation?
- Codeless Test Automation is Irrational
- Automated Testers Advocate for ChatGPT (AI) is like taxi drivers Singing for Driverless cars
- Two Simple Reasons that ChatGPT (AI) Won’t Work in End-to-End Test Automation
- AgileWay Test Automation Formula