AgileWay Test Automation Formula
A proven combination of frameworks and tools for achieving UI test automation success. I have been using the exact formula since 2011.
Non-Medium-Members: now you can view this article free on Vocal Media.
Recently, a former colleague asked me for a successful test automation formula. This happened before; my default answer was to refer him to my book: “Practical Web Test Automation”, and advise him to do the exercises, and then apply them to work. I believe that the success of test automation depends on many attributes: knowledge, willingness to learn and be ready to change, test frameworks/tools, management support, .. etc. More importantly, do it hands-on from Day 1, and every workday onwards. There is no silver bullet.
When I put more thought into it this time, while I still hold the belief that successful test automation has many factors, I believe that there is a good pattern people can follow. A formula, based on my understanding, is a combination of components and processes that has been proven and others can copy and apply to use quickly. The universal applicability makes sense as our target (e.g. web apps) has not changed much.
An important attribute of the formula is completeness. To give an opposite example, I heard this in a meeting at a large financial company: “The DevOps team has been working on trying to run Micro Focus UFT tests in TeamCity for over 9 months. Can someone give us an estimation of the completion date?”. I didn’t remember the reply (from the DevOps team leader) to this, as I was shocked by the question. Apparently, the decision of the CI Server has not taken the UI tests into consideration! This means the people behind do not understand test automation or CI/CD at all, and they are still there.
By following a good formula, inexperienced engineers may avoid common mistakes, such as using Cypress, Gherkins, or JavaScript (as the scripting language).
Projects often make wrong choices on technologies for Test Automation
If a software project chose the wrong technology, correcting it was usually very hard (almost impossible). Reasons are:
- Management generally won’t admit the wrong decision and take responsibility.
- The tech lead will usually defend their wrong approach, and worse, sabotage correction efforts.
- Most IT engineers have never seen a single successful implementation of automated end-to-end testing.
Therefore, a good formula might not guarantee you success, it will prevent costly mistakes and keep you on the right track. Some talented and passionate engineers might be able to pull that off, or wise managers may seek help from a real Test Automation coach.
AgileWay Test Automation Formula
I have a good track record (15 years+) of doing successful Automated UI Testing, which is widely considered difficult. I have used the same technology /tools /practices successfully in many projects. I name it “AgileWay Test Automation Formula”:
- Automation Framework: Selenium WebDriver or Appium (raw)
- Scripting Language: Ruby
- Test Syntax Framework: RSpec
- Source Control: Git
- Testing Tool: TestWise
- Continuous Testing Server: BuildWise, Parallel Testing with BuildWise Agents
Note: The above are 100% freedom, open-source, widely-used frameworks. Even for the tools, you don’t need to pay a cent to get started or forever (just like Zoom, you may use it in free mode forever).
TestWise and BuildWise (in italics) are my choice of tools (disclaimer: I created them, and both were received well internationally), which means they are optional. It is the tech leads’ responsibility to select high-productivity tools. A rule of thumb: manual testers can set up and running tests ~15 mins; CT server can be set up with parallel test execution < 1 day.
Over the last 15 years, I have successfully applied this formula to numerous projects with instant success: at least one key automated test (not simple login) ran in the CT server on the first day. (then added more tests, maintained all, and ran them daily). How could I accomplish that? Easy, because our target (the technologies behind web apps: HTML, JS, and CSS) changed little over the last decade. This means that my formula has worked successfully at Company A, just re-apply it at Company B.
Please note, you have total FREEDOM with this formula, and it can be set up quickly (under 15 mins for developing/executing automated test scripts, another 10 mins for executing them in CT Server sequentially, parallel execution will take a few more hours as setup of agent machines are required).
Automation Framework
The framework that drives the application:
- Selenium WebDriver for testing web apps
WebDriver is a W3C standard, just like all web technologies. So, it is a no-brainer to choose Selenium WebDriver for web testing. Don’t believe some hyped Protractor (dead), TestCafe (hardly mentioned), Cypress (dying, updated: 2023) or Playwright. I have seen many attempts to use each of them, all shared the same outcome: failure (see Definition of End-to-End Test Automation Success). To add another fact, all browser vendors (Google, Microsoft, Apple, Mozilla) support only one automation framework: WebDriver.
Facebook replaces “UI tier” in its Testing Pyramid with “WebDriver”.
Seeing it believing:
“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released.” — DAMIEN SERENI, Engineering Director at Facebook, at Selenium 2013 conference.
Microsoft deprecated Coded UI Test and recommend Selenium WebDriver in 2018.
The above shall be convincing enough, right? I guess some might still think, “That’s because they are tech giants, they have resources to get Selenium working”. Wrong, raw Selenium WebDriver is much easier than Cypress or Playwright, if under proper guidance.
Here is a regression run of the WhenWise suite, consisting of 551 user-story Selenium tests.
WhenWise is just one of several apps that I developed/maintained in my spare time.
Related readings:
- Why Do Most UI Test Automation Fail? (Part 1: Wrong Automation Framework)
- Selenium WebDriver is the Easiest-to-Learn Web Test Automation Framework
- Web Automation Framework Trend ⇒ Selenium WebDriver
- Please, Not Another Web Test Automation Framework, Just Use Raw Selenium WebDriver
- False ‘Selenium WebDriver Cons”
2. Appium + WinAppDriver for testing desktop apps
Appium is also based on W3C’s WebDriver standard.
I have developed and maintained 307 Appium tests for my Desktop App: TestWise, a functional testing IDE.
3. Ruby for testing API/Webservices/Microservices
I don’t use SoapUI, Postman, or REST Assured for API testing. In fact, I have a long history (since 2006) of doing API testing by using the same technology, Ruby.
Related reading:
- eBook: API Testing Recipes in Ruby
4. Appium + iOSDriver/AndroidDriver for testing mobile apps
Though Appium is more well-known for testing mobile apps than desktop apps, I haven’t done real mobile testing (hello-world level tests do not count). The simple reason is that I did not have my own mobile app. I did take a contract role with mobile testing once. It turned out that the mobile app is a wrapper of a web app. This is the area I am planning to explore more in the future.
Based on my experience with Selenium WebDriver and Appium with WinAppDriver, I think Appium + iOSDriver/AndroidDriver will fit well with this formula.
Scripting Language: Ruby
I am proficient in developing Selenium tests in other languages as well.
However, Ruby has proven far better than others. Related readings:
- Why Do Most UI Test Automation Fail? (Part 3: Wrong Scripting Language
- Why JavaScript Is Not a Suitable Language for Real Web Test Automation?
Q & A
- What do you think of using Java with Selenium? Java (and C#) is a compiled language, and I know Java very well (programmed professionally between 1997–2010). However, test scripts shall be written in scripting languages like Ruby and Python.
- How about JavaScript? JavaScript is not a good language for scripting automated tests. Please read Why JavaScript Is Not a Suitable Language for Real Web Test Automation?
- How about Python? If all your team members (including business analysts) are comfortable with Python indentation, it is OK.
Test Syntax Framework: RSpec
A test syntax framework defines the structure in test scripts and provides assertion syntax.
RSpec
RSpec is the most popular BDD framework in Ruby. There are v3.8.0 over 194 million downloads alone. While RSpec is mostly used for integration and unit tests, it is also very suitable for end-to-end tests.
Related readings:
- Why Do Most UI Test Automation Fail? (Part 2: Wrong Choice of Test Syntax Framework)
- Why Gherkin (Cucumber, SpecFlow,…) Always Failed with UI Test Automation?
Please note BDD ≠ “Given-When-Then” (Gherkin).
Q & A
- How about Cucumber (or other Gherkin) frameworks? Avoid Gherkins/Cucumber for end-to-end tests. This is not just my word for it; it is a recommendation from the creator of Cucumber. Please read Why Gherkin (Cucumber, SpecFlow,…) Always Failed with UI Test Automation? (updated 2023–08) Cucumber is dying, my prediction is correct, again.
Testing Tools
A testing tool is a tool that develops, runs, refines and debugs test scripts. A good tool enhances professionals’ productivity without adding constraints (i.e., vendor locking).
TestWise Functional Testing IDE
TestWise is a next-generation functional (dedicated) testing tool that supports Maintainable Automated Test Design and Functional Test Refactoring.
Also, JetBrain’s upcoming testing IDE Aqua is worth a look.
Q & A
- Can I not use TestWise for developing/executing Selenium Ruby tests? Of course, that’s the beauty of freedom (using Selenium). Choose a testing tool that your team members are more productive with. It is worth remembering that the audiences of automated end-to-end test scripts are not just programmers. It’s the whole team.
Continuous Testing Server
With a large automated test suite, the only practical way to manage the test executions of the whole suite is in a Continuous Testing Server. Please note, I don’t use CI servers such as Jenkins as they are only suitable for executing unit tests and integration tests, not UI tests. (As mentioned in My Continuous Testing Journey, I have customized a CT server with some CT features; It is far easier to use CT Server directly).
1. BuildWise CT Server
For the AgileWay test lab, I set up a BuildWise server on a Mac computer (Mac Mini).
2. Parallel Testing with BuildWise Agents on macOS, Linux, and Win10 VMs
I prefer Linux as it is free and fast, while Windows 10 is slow and expensive!
Related readings:
Q & A
- How does the CT server go with unit test execution? CT servers are OK to execute unit tests as well. At AgileWay, we are light on unit testing. Only for complex business logic, we will practice TDD. Maintaining executions of the large UI tests is our main focus. Comparatively, the unit testing execution is too easy.
- How about integration/API test execution?
Yes, we include them in the same way as end-to-end tests, with test script files all starting with
api_XXXX_spec.rb. - Do you separate the tests into sub-suites and run them separately? No, absolutely not. This will cause a number of management issues and confusion. What will you do if 7 out of the 9 suites pass? So, we always run all the tests. If you want to reduce the total execution time, add more build agents. If executions of some test scripts are not reliable, make them reliable. Sorting them into a low-priority suite is not the answer.
- Will BuildWise (and Agents) work in AWS or Azure? Yes, absolutely. The BuildWise server and Agents do not require high-spec VMs. In fact, the lowest-spec (4GB ram) will do.
Test Automation and Continuous Testing are both highly hands-on and practical. If a company's tech leads and managers spent more than one day deciding the formula, their test automation attempts all failed, from my observation. The reason is simple: they don't have the knowledge of test automation and the formula. More discussions or meetings will only lead to a compromise. In the world of test automation and CT, there is only 0 or 1, with little room for compromise.
Further reading:
