A Tale of a Deceptive End-to-End Test Automation Engineer

There is no shame in lacking the ability to perform user end-to-end test automation, but deceitfulness is certainly unacceptable.

The person image generated by ReCraft AI

As a software engineer and test automation engineer, I believe it is essential to emphasize the term “engineer” in our job titles. Engineering, according to widely accepted norms, involves a systematic and reliable approach that aims to create durable and lasting solutions. For instance, if a visually appealing bridge were to collapse merely three days after its inauguration, the “civil engineer” responsible for its design would not only lose their professional qualifications but could also potentially face legal consequences, such as imprisonment. However, the software industry often exhibits a more lenient attitude towards such matters. Therefore, there is a high percentage of fake automated testers.

Long-time readers have witnessed me citing world-renowned agile and testing experts multiple times, these experts have conveyed the same message: implementing genuine and valuable end-to-end test automation via the user interface (UI) is an exceptionally challenging endeavour. Few succeeded.

“In my experience, great developers do not always make great testers, but great testers (who also have strong design skills) can make great developers. It’s a mindset and a passion. … They are gold”. - Patrick Copeland, Google Senior Engineering Director, in an interview (2010)

“95% of the time, 95% of test engineers will write bad GUI automation just because it’s a very difficult thing to do correctly”. - this interview from Microsoft Test Guru Alan Page (2015), author of “How we test software at Microsoft”

“Automated testing through the GUI is intuitive, seductive, and almost always wrong!” - Robert C. Martin, co-author of the Agile Manifesto, on his blog (in 2009)

Sadly, many software professionals, due to their limitations, did NOT have this realization. Quite a number of people thought: “Test Automation is just record-n-playback, easy”. How wrong is that?

“Testing is harder than developing. If you want to have good testing you need to put your best people in testing.” - Gerald Weinberg, software legend, in a podcast (2018) (current generation of programmers might not heard of Gerald Weinberg, check out the a tweet by DHH on 2023–12–27: “Nobody taught me more about how to do that than the late, great Gerald M. Weinberg.”)

BTW, I have been achieving E2E (via UI) test automation success (enabled real DevOps, such as Daily Production Releases) for over 15 years. As an engineer, I say things with proof, here it is, Showcase a 500+ End-to-End (via UI) Test Suite: E2E Test Automation is Surely Feasible for Large/Complex Apps.

In this article, I will present a tale story, illustrating a typical fake automated tester when exposed.

Table of Contents: · 95+% E2E Test Automation Engineers I met were Fake · A Tale: Test Automation Engineers Should Take Responsibility for Their Work · Why people like Toby is common?

95+% E2E Test Automation Engineers I met were Fake

Don’t get offended if you are currently doing end-to-end test automation. I haven't met you, so you might well be in that 5% (or 0.25% in Alan Page’s judgement. Compared to Alan’s, my figure is quite conservative).

Here is a story told by Michael Feathers (renowned agile expert and the author of ‘Working Effectively with Legacy Code’) in 2009:

It happens over and over again. I visit a team and I ask about their testing situation. We talk about unit tests, exploratory testing, the works. Then, I ask about automated end-to-end testing and they point at a machine in the corner. That poor machine has an installation of some highly-priced per seat testing tool (or an open source one, it doesn’t matter), and the chair in front of it is empty. We walk over, sweep the dust away from the keyboard, and load up the tool. Then, we glance through their set of test scripts and try to run them. The system falls over a couple of times and then they give me that sheepish grin and say “we tried.” I say, “don’t worry, everyone does.”

I also have seen the above (the difference is that I did not ask, instead, checking the test execution reports) many times, and the situation hasn’t changed.

Now a question, do you agree that “the team (in Michael’s story) is doing fake test automation”? Of course, it is. If they (team members) were honest, they would just say “We tried and failed” first, not pointing out the ‘test machine’ and trying to run the tests (and failed a couple of times, betting on luck?). Only then, acknowledged they did not have the capability.

Why? Because the team feel embarrassed to acknowledge they couldn’t do end-to-end test automation, they lied.

In my opinion, not being to do real test automation is nothing to feel shameful (see reasons below), but lying is.

A Tale: Test Automation Engineers Should Take Responsibility for Their Work

By principle, everyone would agree with the above statement. However, in test automation, this is usually not the case. Let me elaborate with a hypothetical (but typical and reasonable) story.

Toby, a senior test automation engineer (contractor), has worked on the projects for 5 months. During the stand-ups, he reported done testing for a number of user stories, and Jira shows that too.

Russ, a new project manager, joined the team (staff change). Russ came to Toby, “Hi Toby, the CIO has been trying to do ‘release early, release often’, as you know, end-to-end test automation plays an important role, in regression testing”.

Toby nodded, “Surely it is”.

Russ said, “Good, we have the same understanding. Now, I am planning to do full regression testing. Then we may get an idea of how often we can do a release. Of course, we have to take into consideration a few rounds of regression testing, to retest bug fixes.”

Russ continued, “You are the only test automation engineer in the team. Can you get a list of user stories that your test automation suite cover? I will tell the manual testers not to test those, as it is covered by your test automation. OK, just simply send me a recent test report”.

Toby replied, not confidently, “OK”.

After Russ left, Toby rushed to open the test execution in the so-called CI/CD pipeline. Under UI test automation, the most recent run was 3 months ago, and over 50% of test execution failed.

The next day, Toby came to Russ’ desk and said “The automated tests ran fine at the time of creation. Because the app and test data changed, a number of automated tests failed. I might need a few weeks to rectify them”.

Russ was unhappy and asked “I thought maintaining the automated test suite is part of your job. Don’t you agree? The app will always change, we are doing Agile, right? Can you get it done in three days? And from that on, make sure all tests are well-maintained”.

The next day, Toby came to see Russ with a smile. He said, “Russ, I found a better test automation framework: Playwright, which solved the issues and limitations of our current framework: Cypress. I am confident that I can achieve migrating Cypress to Playwright in three months.”

Russ was not impressed, “To my knowledge, it was you who suggested Cypress, to replace the previous failed attempt using TestCafe, 5 months ago. If Cypress was no good as you are saying now, why did you select Cypress in the first place then? ”

Toby was in silence.

Russ continued, “Our app is just a standard web app, from what I see, unchanged for over 10 years from a testing perspective. I have once seen one real Agile project where the raw Selenium WebDriver test suite enabled ‘daily production releases’. That test automation engineer implemented a handful of core business flow tests in Selenium and ran them in a CT server on the first day, then every day onwards.”

Facebook’s Testing Pyramid, because it uses WebDriver for End-to-End UI Testing, so calls the “WebDriver” tier. Check out “Continuous Integration at Facebook” video on Youtube.

“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released.” — DAMIEN SERENI, Engineering Director at Facebook, at Selenium 2013 conference.

Toby said, “It is my mistake for using Cypress, frankly, not just me, many other testers believed …”

Russ interrupted him, “You were hired to do useful testing, not experimenting. You are free to dump Cypress to use another, in whatever language. As the project manager, I just want to see real end-to-end execution and test results, daily. Regarding the language, just as an observer, I prefer not JavaScript for a simple reason, to my knowledge, this (Cypress) is the third consecutive test automation failure with JS-based frameworks, the previous two were Protractor and TestCafe.”

Toby said, “JavaScript is the most popular ….”

Russ stopped Toby and said: “I, and our project team do not care about which language is used for end-to-end scripts at all, as long as it works well, providing real value. You said ‘Yes’ to my request yesterday and now just proposed a different framework. Anyway, in three days, our team members expect to see the daily e2e test execution reports, regardless of the framework or language. What matters is: automated end-to-end test scripts can verify functionalities as an end-user, daily, and remain valid while the app keeps changing.”

Three days later, Toby resigned. In his Résumé, an entry was added: “Successfully implemented end-to-end test automation at Company X using Cypress”.

What happened to the project’s test automation afterwards? Russ quickly hired a passionate IT graduate, Darcy, who has no testing or automation experience. He engaged one Test Automation Coach (the test automation engineer he mentioned before) to train Darcy. After one-day training, Darcy started working on converting Cypress tests to raw Selenium WebDriver + RSpec, under the coach’s mentoring. From the 2nd day, the team started to see the e2e test reports on the CT Server (like the one below), detecting regression issues.

A run of WhenWise UI Regression Suite, consisting of 569 raw Selenium WebDriver + RSpec tests.

Within the next 5 working days, all Cypress tests (not many anyway) were converted (with the help of BA, as many Cypress tests were invalid or couldn’t complete) to a much better version using raw Selenium WebDriver + RSpec, which BA and manual testers can read and run comfortably. At least two automated regression testing reports each day. Of course, Darcy and the coach had to make necessary updates to the test scripts due to recent changes that were checked in and impacted the automated test scripts. Thanks to the Maintainable Automated Test Design and good tools (e.g. TestWise and BuildWise), maintenance was managed effectively.

The coach told Russ, “Darcy learned well, I don’t need to be here full-time.”, he turned to Darcy, “Remember, running the whole test suite every day at least once. With the growing tests, you will find it harder and harder to maintain them valid. If you are unable to get all tests valid (PASS or failed with good reasons) for 2 days in a row, get Russ to contact me.”

The made-up story comprises a blend of my firsthand witnessed experiences, including the ‘resignation’ part and “Darcy learned to do useful testing in one day”.

Here I want to point out one thing many people would miss: the importance of Russ, a real Agile manager. Most ordinary project managers, while claiming Agile, would tolerate the faking end-to-end test automation behaviour. Why? It is easy to do so as it is so common. Once test automation is real, while certainly sounds wonderful, but will bring unknowns, i.e. fears to some. Good IT managers, who understand the importance of End-to-End Test Automation, are rare, too.

“One Team One Dream” vs “Another Day Another Dollar”

Do you think Toby did a satisfying testing job? Of course, not. Will the ‘automated tests’ he left with will be of any use? Highly doubtful. In other words, Toby is a fake test automation engineer, at least for that project. Fake automated testers ruin the reputation of test automation engineers.

Some would argue, “It might be a little harsh to call Toby a fake test automation engineer. When first testing the assigned user story, Toby might have found some good defects”.

Yes, that could be true. But, a manual tester could have accomplished that, most likely, a lot quicker (and also cheaper, as manual testers usually get paid less). Please don’t forget we are talking about the expectation of a Test Automation Engineer.

Engineering: proven and repeatable process that builds to last.

The real power of end-to-end test automation is regression testing, which means, the automated tests are reliable to run frequently on a daily basis, especially towards to end of the development cycle. Now, if someone tells you (as the project manager), that created the automated tests worked only for that day. How will you feel?

Why people like Toby is common?

Creating automated end-to-end tests is very easy, compared to maintenance. Check out my article, “Test Creation Only Accounts for ~10% of Web Test Automation Efforts”.

To be fair, it is largely not Toby’s fault. I am not belittling ‘fake test automation engineers”, because our IT education (university level) never included End-to-End Test Automation. Also, few companies provide training and seek external help such as real test automation coaching. At the same time, there is a huge demand for test automation as most software projects claim “doing Agile”. The responsibility lies with the management for failing to adequately evaluate them and offer comprehensive hands-on training.

https://www.linkedin.com/feed/update/urn:li:activity:6929785344288546816/

Imagine Toby confessed no test automation skills, and showed a sincere desire to learn. Thanks to Russ, he would access the real training and the opportunity to work with and learn from a real test automation coach. What a pity!

But I do despise ‘fakers’ who continue to do fake stuff after witnessing real test automation because faking is easier. These people ruin the reputation of test automation engineers.

In a future article, I will show anyone (with honesty) can do useful end-to-end test automation quickly.

Related reading: