Test Automation and Continuous Testing Competition Week
A Practical and Fun Way to Create a Successful Test Automation and Continuous Testing Formula in Your Company
Non-Medium-Members: you can view this article free on Vocal Media.
Most software companies (or IT divisions) have no clue how to do automated functional testing or execute these tests in CI/CD. Not a believer? Check the end-to-end test reports in your CI server. Without executing automated end-to-end tests, there will be no ‘Delivery’ (of CD), right?
I am talking about the really useful automated testing that provides the teams useful and real feedback which will ultimately enable the software to “Ship early, ship often”. For those companies who have claimed that they are doing automated functional testing, what they have in common is that there are dozens of different non-working test automation frameworks/tools existing within the projects and the company. Subsequently, the outcome is the same: the automated testing is either failed already or is about to fail.
I still remember a meeting that was called by a newly-on-board test architect in a large organization. There were about 70 testers in the room. The test architect asked a representative from each team to shout out their test automation framework or tool. He was surprised that there were so many different frameworks and tools that were currently in use within the organisation. He was even more surprised to learn that the testing was still mostly done manually, while with all these test automation frameworks and tools!
It was obviously a mess and a big waste, but it is very common. How will you solve it? In this article, I will suggest a practical (and fun) way to create successful test automation and Continuous Testing (CT) Formula for your company, in a matter of days. I call it “Test Automation and CT Competition Week”.
With the executives’ support, it is actually quite easy and cost-effective. Here is how: set up a competition event (like some companies’ innovation day) and invite all interested engineers to participate in groups/teams.
1. Nominate a set of acceptance tests for your main app. 25 is a good starting number.
Usually, a company has already defined a suite of acceptance tests (for manual regression testing) for its main application. Therefore, it shall be easy done. My suggestions for test selections:
- Cover a broad range of business functions
- Web features in testing perspective, such as AJAX, verifying data in a complex table, verifying generated PDF, file upload, frames, multiple browser tabs, multi-domains, drag-n-drop, pages with dynamically-generated IDs,…, etc
- A good mixture of simple, medium, and complicated test cases
The number of tests must not be too many or too few. Too many, people tend to give up; Too few, besides lacking the coverage, will be inadequate to verify the CT process.
2. Announce Test Automation and Continuous Testing Competition (3-5 days)
Some companies might call it “innovation day or week”. The movie “The Internship (2013)” had this kind of competition within the intern teams at Google. The ideal duration of the competition is 4 days (leaving some time for introduction and reviewing). It should definitely not exceed one week.
The purpose: each team will be given exactly the same task to complete a regression suite
A team is consists of 1 business analyst or manual tester and 3 software engineers in Test or Software Engineers. The business analyst or manual tester in the team provides business knowledge of the test cases.
The rules of competitions shall be simple:
- No limitation of technologies (important!)
- At least 3 new functionally correct automated tests are added to the CI build daily. If failed to meet this goal for 2 days, the team will be disqualified.
Here I would like to highlight the importance of “No limitation of technologies”. Executives don’t usually know a sad fact: many good ideas were shot down by the tech leads and mid-level managers due to their fear (of something new or they don’t understand).
3. Prepare the environment and support
Before the competition starts, ensure the teams get all the infrastructure support they need, such as
- the testing server can handle the load
- dedicated test user logins
- testing tools (trial licenses are fine)
- virtual machines for parallel testing
- all access privileges are sorted out
This is also a test for your infrastructure team.
3. About halfway into the competition, introduce some simple application changes that would affect some tests in the suite
The purpose is to check the team’s capability to maintain automated test scripts. I suggest the following changes:
- Changes to some HTML element IDs and names
- Addition of new mandatory elements, e.g. text field, on some pages.
- Removal of web elements
- Changes to some button and link texts
- Extend some AJAX operations
- A brand new form is added to the business process
The software updates (typically, check out a tagged revision from the Git repository) shall be rolled out gradually. Taking a 5-day competition as an example, there should be releases on the early mornings of Day #3, #4, and #5.
4. Introduce some defects to the application
This is to verify the quality of regression tests and how quickly the teams can detect them.
5. Judge objectively and solely based on the results.
The judgment criteria:
- Completion of the target test suite.
- Correctness.
- Execution speed.
- Detection rate for the known effects
- Execution reliability in CI/CT server. The overall pass/fail rate is not a good criterion. A better way is to check whether a green build can be obtained (pass all tests) at the end of the day.
- The quality of test scripts. A simple way to judge is to check whether the business analyst or manual tester understands the test scripts.
- The finishing time. On the comparable results, the team that finishes the test suite earlier wins.
Reward the winner!
The company surely needs to put into some investments for this competition. The cost, relative to the company’s gains or the money it would save, is minor.
Two outcomes resulted from the competition are:
1. A successful formula for test automation and CT have been found
A big win for the company is that the success will boost the confidence and grow the capability on this ‘proven’ track. I can share some experiences from companies that do not have these open exercises.
- The tech leads and test manager decided on Cypress but found Cypress did not support iFrame a week later. (Cypress and some other JS-based automation frameworks have limitations. Check out this article: Why Cypress Sucks for Real Test Automation? Part 2: Limitations)
- Cucumber was sold by ‘fake agile coaches’ to be used in testing. An agile coach boasted that he had done over 200 Cucumber end-to-end tests in several of his past projects. In reality, this test automation team (with the involvement of this ‘agile coach’) has never been able to deliver a successful test report over 12 Cucumber tests. The team was struggling with maintenance from Week 2 and onwards until these tests were aborted. (Check out this article: Why Gherkin (Cucumber, SpecFlow,…) Always Failed with UI Test Automation?)
As we all know, after a decision was made, the management who made this decision (no matter right or wrong) often chose to defend it. That’s why so many companies tend to renew the licenses of testing tools though the companies do not actually use for real work.
Besides, the company gets a regression suite of 25 automated tests which are useful now.
2. No single team delivered satisfactory results
This is probably the likely outcome for most software companies.
Though it means the company does not have the capability, still a valuable finding for the company.
Then review and act based on the feedback:
- identify what approaches definitely won’t work, such as record-n-playback and use of Gherkin syntax.
- identify the limitations of certain technologies, such as Cypress does not support two browser windows/tabs, …, etc.
- prevent many unnecessary (and costly) attempts from individual teams.
- improved awareness of infrastructure support for real Continuous Testing.
- improved awareness of test automation beyond QA. For example, Business analysts may want to run automated tests but are uncomfortable with the complexity of the tools.
- improved respect for CT engineers’ hard work
- help some engineers to give up fixation on wrong framework/tools. Meetings won’t help, I tried. Once, most of team members acknowledged I was a better Java programmer, but I couldn’t convince them to use Ruby to write automated tests after numerous discussions.
- …
If necessary, the company might do another competition event in a month time. It is still no luck, engage a real test automation and CT coach to help. Now the executive’s decision to seek external help from a real test automation coach might face much less obstruction.
Of course, the company can use the same competition process (with a shorter timeframe) to test whether a test automation coach candidate is fake or not.
Relative readings:
