Showcase a 500+ End-to-End (via UI) Test Suite: E2E Test Automation is Surely Feasible for Large/Complex Apps
Implementing End-to-End Test Automation, the most valuable type of testing, is most QA Engineers’ responsibility. Be a ‘test automation engineer’ rather than a ‘test automation coward’. Don’t have the capability? acknowledge and learn! I hope this showcase boosts some test automation engineers’ motivation to do real E2E Test Automation.
Last night, I saw this in my LinkedIn feed.
The view of “Avoiding End-to-End tests” is common, and I usually ignore it. However, this one mentioned the specific number, promoting should do 5 instead of 500 E2E tests. I posted a link to my article, “WhenWise Regression Test Suite Reaches 500 Selenium tests and ~300K Test Executions” (2021).
Table of Contents: · A Tester challenged me, “WhenWise might be just a simple demo site” · WhenWise E2E Test Suite Stats ∘ FAQ · A few WhenWise Features and its E2E Testing ∘ 1. User Authentication related ∘ 2. Client Booking (one on one) ∘ 3. Business Manage Booking on Calendar ∘ 4. Business Settings ∘ 5. Customer Payment · Demonstrations · WhenWise is Feature-Rich and Real
A Tester challenged me, “WhenWise might be just a simple demo site”
To my surprise, one tester started to question the complexity of the WhenWise app. He knew the WhenWise practice site, why not just click the link, and log in with the provided username/password? (quite logical for a software tester, right?) Then He would know in seconds. Instead, he is making a fool of himself. A typical behaviour of fake testers: like to talk, don’t like hands-on doing.
1. Is that app a demo website meant for practicing test automation 2. How many real users does your application have? 3. How many features do you (as a developer) release on this application? 3. How complex is your underlying system? 4. What does the system architecture of this entire application look like?
If the answer to these questions does not match with a platform like Facebook/AirBnB/LinkedIn/Spotify or any other similar exponentially complex platform out there in the world then you are comparing apples and oranges.
Let’s talk about real world systems. That’s where the real problems w.r.t flakiness happen with e2e tests.
In short, he did not believe that I could maintain a suite of 500+ E2E tests for a complex app. He is wrong! WhenWise is a feature-rich app (which I will prove in a moment), besides WhenWise, I managed the E2E test suites for ClinicWise, TestWise (a testing IDE, demonstrated at Agile 2009 Workshop), BuildWise (an international award-winning CT server), SiteWise CMS, .., etc, mostly in my spare time.
Be aware of this kind of person, who tried to bring up deceiving questions (muddling the water) to avoid the tasks at hand, in order to cover his incompetence. There are some obvious errors in his logic, in the context of the post ( “5 E2E Tests instead of 500, due to flakiness”).
In real Agile, every user story should be covered by at least one automated End-to-End (via UI) test, otherwise, it is not “Done, Done”, Fake Agile. it is simplely a matter of skills and capability.
According to him, because the target app is feature-rich as Facebook/LinkedIn, then just do 5 E2E tests; If the target app is relatively simple, can he manage 50 E2E tests (Level 2 of AgileWay Continuous Testing Grading)? I highly doubt it, the extremely few test automation engineers who can actually achieve Level 2, definitely would not say something like the above.
- Most web apps face E2E test automation challenges in a similar way. This is obvious, the Web is the same (we automate the operations that a pre-schooler can perform on Chrome). The App’s system architecture is largely irrelevant, that’s why it is “Black-Box Testing”. On consulting, based on what customers show me, I implement real test cases live.
- The real test automation challenge lies in maintenance, while the application changes frequently. Maintenance effort goes exponentially with more tests, the flakiness rate will go up, but should always remain extremely low at the test script level (otherwise, how could you a green run?). Flakiness in E2E tests is mostly due to the tester’s capability. I know that because I put a lot of effort into making E2E test reliable. Check out my article, Working Automated Test ≠ Good Reliable Test
- The number of users in a system will impact load testing mostly, not necessarily on features, it s load testing vs functional testing.
- Regardless your app is large/complex or small/simple, a sizeable (definitely not 5) Automated E2E test suite provides great value. After all, it is the foundation of Agile.
The real problem is that few automated testers (see Alan Page’s estimation below) can do proper E2E UI Test Automation, which means running a suite of E2E tests daily as regression testing. In my city, I haven’t yet seen a good implementation with over 50 tests (except for the test automation I led).
“For 95% of all software applications, automating the GUI is a waste of time. For the record, I typed 99% above first, then chickened out. I may change my mind again.” — Alan Page’s Blog (2008), author of How We Test Software at Microsoft
“95% of the time, 95% of test engineers will write bad GUI automation just because it’s a very difficult thing to do correctly”. - this interview from Microsoft Test Guru Alan Page (2015)
Because there are so many fake ‘automated testers’, they would find all kinds of excuses. A Simple Question from a management perspective will reveal that:
“If a QA Engineer can only do 5 E2E Tests, Why should I hire him?”
Some would argue, “We are doing API testing, Cypress Component Testing or Visual Testing”. Don’t be deceived by those comments. A simple fact.
“Watir, Selenium WebDriver, WebDriverIO, PhantomJS, ProtractorJS, TestCafe, Puppeteer, Cypress and Playright, all of those were created for end-to-end test automation, driving web apps in browser”.
Fake automated testers, the fact hurts, right? As an owner of several software products, I definitely won’t keep these kinds of people around.
As “The World Quality Report” pointed out, product owners care mostly about End-User satisfaction, which translates to E2E Testing via UI in the context of software testing, right? Avoiding it?! That’s being a coward.
I mentored many testers, and I told them on the starting day: DO or LEARN. DO means hands-on working on test automation scripts with quick results (in minutes or hours), as the whole suite will be run daily as regression testing. Work is work, we don’t experiment with different frameworks here, check out my article, “How do I Start Test Automation on Day 1 of Onboarding a Software Project?” There are a considerable number of fake automated testers who felt so uncomfortable with hands-on scripting/maintaining and opted to do manual testing.
In this article, I will show some components of WhenWise and share some of my testing tips along the way.
My purpose here is to boost the confidence of QA Engineers: E2E Test Automation is feasible for complex apps, such as WhenWise. Even TestWise IDE, (this got to be considered large and complex, right?), which has 330 Appium + WinAppDriver E2E tests. For me, comprehensive E2E Test Automation is a must, as I do “Daily Production Releases” for all my apps.
Whether you can do real E2E Test Automation is simply a matter of capability, you can learn. Is it shameful to raise a ‘white flag’ easily and still call yourself ‘QA Engineer’ or “Test Automation Engineer”?
WhenWise E2E Test Suite Stats
Automation Framework: Selenium WebDriver
Test Syntax Framework: RSpec
Scripting Language: Ruby
Test Count (as of today, 2023–08–10): 569
Test Steps: 29,845
Duration: 5 and half years, from 2018–01–24 (the date of first execution in BuildWise CT Server) to present
Do you really maintain and run these E2E all those years? Yes.
Total Test Execution: 359,158
FAQ
How is it created? Developed in TestWise IDE
How is it run? Executed in the award-winning BuildWise CT Server
How complex is the WhenWise app? See below, I can give a number, though, 193 web pages or modals. How did I know? because E2E tests follow Maintainable Automated Test Design, using page classes. A simple Ruby script gets me the info.
How Long does it take to run the whole suite? ~50 minutes on 5/6 buildwise agents (run in parallel), ~3 hours 40 minutes if one machine.
Do you really run this suite (569 Selenium RSpec tests) as regression testing? Yes. In fact, I just did two a couple of hours ago. Here is a recent execution report on the BuildWise CT server (note the time stamp).
Do you really push a build to WhenWise production after getting a green run of this E2E test suite? Of course, I have been doing this for all my apps for over a decade. Yes, just released a new version of WhenWise. Otherwise, How could I manage developing/maintaining all my apps (in my spare time)?
What’s the major benefit of this E2E suite? Huge development productivity boost, WhenWise is just one of several apps I created (solo) and maintained in my spare time. Also, I do “daily production releases” if any code changes on that day. E2E Test Automation enabled all of that, plus a lot more.
A few WhenWise Features and its E2E Testing
Below, I just quickly show a few components of WhenWise and its E2E testing.
1. User Authentication related
WhenWise supports sign-up and login with Facebook and Google accounts. For testing, two seeded users (Facebook and Google) were used to emulate sign-up/sign-in with Facebook/Google.
Of course, there are usual E2E tests:
- User Sign-up with email verification Tip: Use a Fake SMTP Server such as MailCatcher to verify emails.
- Sign in with “Remember me”
- Forgot password
- Change password, Change User Profile, …
- Verify access privileges
Don’t underestimate those common tests (which almost universally existed tests for any web app), the majority of so-called ‘automated testers’ got it wrong, i.e., flaky tests.
Check out my most-liked article on Medium, One Simple Test Automation Scenario Interview Question that Most Candidates Failed. In other words, every so-called automated tester and senior test automation engineer I interviewed, lacked a basic understanding of real test automation, even this simple test they designed was flaky. (in real work, they often blame the framework)
2. Client Booking (one on one)
A client finds a business and makes a booking (on mobile, iPad or PC).
Readers with coding experience would know, this is a highly-dynamic page, and utilize a lot of JavaScript. Of course, there are many E2E test cases created for this feature. How did I make E2E tests (for dynamic) not flaky at all? Not auto-waiting (which can still cause flakiness), check out my article, Test AJAX Properly and Efficiently with Selenium WebDriver, and Avoid ‘Automated Waiting’
Please note, the screenshot shows a booking for driving school, so there are custom data such as Car type and Pickup address (with Google map validation). Yes, because of the WhenWise E2E regression suite, I am free to customize our booking screen to suit different business types.
Besides one-on-one booking, there is another booking type: Group Lesson (for Yoga classes alike), which offers a completely different booking screen and business logic.
Experienced developers would know that is not easy to get two business flows into one. The E2E tests prevent many regression issues.
3. Business Manage Booking on Calendar
A Business user views and manages booking on the Calendar.
As you can imagine, many features here, therefore many E2E tests, such as move, delete, adjust, filter by professionals, different views, …, etc.
Calendar testing is often known as flaky ones, why? The test data depends on the execution time. Yesterday’s tomorrow is not Today’s tomorrow. When I designed the tests, I was extremely careful with that to make sure my tests are 0% flaky regardless of execution time, morning, Monday, Sunday, end of month, and Timezone switching clock. You get it, without skills and endeavour, these tests will be flaky.
Here is the interface to create an internal event or booking (if the client management module is turned on).
The modal is highly dynamic, again, a lot of E2E tests here to provide the safety net.
4. Business Settings
To get a quick view of how many features an app is to see its setting page. WhenWise provides many business settings.
Each setting leads to one or more E2E tests.
5. Customer Payment
WhenWise has quite standard customer plans.
Accept credit card payment (via Stripe)
Stripe has a testing mode, our E2E tests will utilize that.
Besides the pre-defined plans, customers can purchase extra add-ons for $1 per resource per month.
Obviously, this means many business features, thus, very good E2E Test coverage for them, too.
E2E Test Execution Demonstrations
There are a lot more business features, such as SMS, QRCode, Service Notes, Invoice/Payment, Reviews, Coupons, Landing Page, Reporting, Charting, Internalization, …, etc, in WhenWise. I don’t think I need to go further to convince WhenWise is a feature-rich booking platform.
Let’s see some E2E test execution videos.
WhenWise is Feature-Rich and Real
Yes, we have customers (including my own company) running their business on WhenWise, it is real and has been running for over five years.
My daughter had the idea of WhenWise when she was in high school and did the first implementation. I made it a commercial product.
One sentence on my daughter’s Résumé: “I booked my first driving lesson using the app I created!”
You actually can sign up (as a user or a business) and use it today (starting from the free plan for business).
I created another sandbox WhenWise server (the same code base), Free Test Automation Practice Site with Database Reset for motivated automated testers to learn and practice test automation. With a database reset feature implemented, it is much easier.
Also, I offer complimentary coaching for TestWise customers, who can replicate the test automation challenges on the sandbox WhenWise site. I will share my solutions for free. The reason I can do that: my E2E tests are always valid.
Again, I hope one automated tester, after reading this article, gets the motivation to take on real test automation. Live is too short for doing fake work.
Start today, get tests (even starting with just 1) running in a CT server of your choice, then run the whole suite every day. Check out my daughter’s guide, Set up, Develop Automated UI tests and Run them in a CT server on your First day at work.
Maintain all tests to be valid at the end of each day. Along the way, you will understand more and more about real test automation and Continuous Testing, you will come up with better designs and solutions. That’s engineering and growing.
Now you know there is at least one guy (besides Facebook, see below) who can manage a 500+ End-to-End UI test suite (using raw Selenium WebDriver) with relative ease.
“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released.” — DAMIEN SERENI, Engineering Director at Facebook, at Selenium 2013 conference.
So stop saying ‘Do fewer E2E tests as they are flaky’, and be a ‘test automation engineer’ rather than a ‘test automation coward’.
Related reading:
- My eBooks: - Practical Web Test Automation with Selenium WebDriver - Practical Continuous Testing: make Agile/DevOps real - API Testing Recipes in Ruby You can the above books (plus others) free by joining Medium Membership before 2023–09–01, via this referral link. Once you become a Medium Membmer, you can read any content (not limited by the ones by me) on Medium.
- Flaky Tests in Test Automation Clarified — Part 1: Flaky Tests are Common but people tend to Neglect them
- Recommend a Great CI/DevOps Presentation: “Continuous Integration at Facebook”
- One Simple Test Automation Scenario Interview Question that Most Candidates Failed
- Benefits of Real Test Automation and Continuous Testing series: Executives, Managers, Business Analysts, Developers, Testers and Customers.
