avatarZhimin Zhan

Summary

The article argues that Playwright's parallel execution support is misguidedly presented as an advantage over Selenium, contending that such a feature should be managed by a Continuous Testing server rather than the automation framework itself.

Abstract

The author of the article challenges the common belief that Playwright's native support for parallel execution is a significant advantage over Selenium. The article posits that parallel execution is not a core responsibility of an automation framework, which should instead focus on reliable web element control and intuitive syntax. The author cites Facebook's successful use of Selenium WebDriver for continuous testing without relying on parallel execution within the framework. The article also demonstrates Playwright's parallel execution capabilities, noting that while it can reduce test execution time, it introduces practical limitations such as resource contention and test conflict issues. The author, who has experience in implementing parallel execution in continuous integration environments, advocates for leaving parallel execution to specialized Continuous Testing servers, which can manage such tasks more effectively and are framework-agnostic.

Opinions

  • The author believes that the core function of an automation framework like Selenium or Playwright is to ensure reliable and intuitive control of web elements, not to handle parallel execution.
  • It is suggested that Selenium Grid, despite being a part of the Selenium suite, is a poorly designed tool for parallel execution, and its shortcomings are detailed in a separate article by the author.
  • The article claims that Facebook's Continuous Testing server, Sandcastle, effectively manages parallel execution using Selenium WebDriver, demonstrating that an automation framework does not need built-in parallel execution support.
  • The author argues that parallel execution support within Playwright is not as beneficial in real-world scenarios due to limitations such as computer resource competition and test execution conflicts.
  • The author emphasizes that many end-to-end tests are not compatible with parallel execution due to potential conflicts, such as simultaneous login attempts that can cause test failures.
  • The article promotes the use of a Continuous Testing server for parallel execution, citing the author's own experience with creating such a server and the benefits of using real machines as separate build agents for testing.
  • It is highlighted that a Continuous Testing server can provide advanced features beyond the automation framework, and that parallel execution should be one of these features, as it is framework-agnostic and can work with various automation tools.

Correcting Wrong ‘Playwright’s Advantage over Selenium” Part 2: “Playwright has Parallel Execution Support”

An automation framework should NOT concern parallel execution, which falls under a Continuous Testing server’s responsibilities.

The first article of this series was immediately awarded “Boost” status by Medium creators. I will continue to correct the second wrong claim, “parallel execution support”, in this YouTube video, “Playwright vs Selenium: What Advantages Make Playwright the Winner in Automation Testing Battle 🏆”. Again, like all his other claims, this presenter is wrong, this article explains.

This article series:

Table of Contents: · Claim 2, “Playwright has Parallel Execution Support”. (It's better without it) · Parallel Execution Support is a Continuous Testing Feature and shall not be in the automation framework. · Parallel Execution in Playwright in a perfect world. · The Problems with Playwright Parallel Execution · Parallel Execution in the real world

Claim 2, “Playwright has Parallel Execution Support”. (It's better without it)

Playwright fans might think, “Zhimin, your first article may be valid, but surely, you can’t argue with this, as Selenium WebDriver does not come with parallel execution”.

It is not strictly true to say Selenium WebDriver does not offer parallel execution. Selenium Grid has always been a part of the Selenium suite. But Selenium Grid is more a complimentary tool and a badly designed one. For reasons, see my article, Why I Don’t Use Selenium-Grid?

The core Selenium framework does not have “parallel execution support”, in my opinion, is an advantage over Playwright. “Less is more” is more true when the ‘more’ is wrongly implemented.

Parallel Execution Support is a Continuous Testing Feature and shall not be in the automation framework.

Parallel Execution is a Continuous Testing feature and should NOT be in the automation framework. As a web automation framework, the core is to ensure that web elements are driven reliably in an intuitive syntax and fully support the web technologies defined by W3C. Selenium WebDriver accomplishes this task well.

Not Convinced? Let me show a quote and an image by Facebook.

“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released.” — DAMIEN SERENI, Engineering Director at Facebook, at Selenium 2013 conference.

from this great presentation, “Continuous Integration at Facebook

Facebook’s Continuous Testing Server Sandcastle handles the Parallel execution. Apparently, Facebook achieved parallel execution extremely well, using Selenium WebDriver. Please note the year.

Some might argue, “That’s Facebook, and it seems Sandcastle is not available publicly. What’s the point of talking a secret tool that we cannot use?”. You can use BuildWise, a free, open-source and international award-winning CT server.

Parallel Execution in Playwright in a perfect world.

Let’s quickly visit Playwright execution. First, let’s start with the sequential mode, called one worker in Playwright.

> npx playwright test  --headed --workers 1 --reporter=html

A sample test report:

Run a small suite with five workers:

> npx playwright test  --headed --workers 5 --reporter=html

As expected, I saw five Chromium processes (shown on macOS Dock).

The test report:

The time difference between 1 and 5 workers is not big: 35.5 vs 18.6, a saving of 47.6%.

Of course, this is not a thorough benchmark (as it uses a very small suite of stateless test scenarios). But, it shall provide a Playwright parallel execution in a nutshell.

  • It looks good for demonstration.
  • It might leave the audience thinking that the long execution time problem could be resolved by simply increasing the worker size.

However, astute readers will know (from the above) that parallel execution support is not much useful in real life.

The Problems with Playwright Parallel Execution

The above Playwright Parallel Execution demonstration might impress many people. However, it is not good for real testing situations.

1. Workers are Competing for computer resources

Below is the activity monitor (on macOS) showing the CPU usage during the test execution.

Each Chromium process (started by Playwright) consumed ~16% CPU. In other words, the worker count you can use is quite limited.

2. The demo test suite I used is super simple and fast.

The test scripts in my demo are simple and quick (~2 seconds for one test case). When testing an enterprise app, e.g. Guidewire or Microsoft Dynamic 365, a typical E2E scenario will take about 5+ minutes. (The audience won’t have the patience for the demo, anyway).

With proper testing cases, the browser (chromium) processes will consume considerably more computer resources, e.g. CPU and Memory. Also, my simple demonstration shows the benefit of parallelism was not that great: saving 47.6% with five workers.

3. Many tests won’t run well together.

The more important reason is that many E2E tests won’t run well in parallel, i.e. conflicts. Let me illustrate with a simple scenario. You run the following two test scripts in parallel:

  • User login (with username ‘manager’)
  • User change password (with the same user ‘manager’)

Let’s say you have designed the later test script without side effects (most so-called ‘automated testers’ don’t have this knowledge; see the article below).

What will happen when both tests are running and the second test starts before the first one? The first test might fail because the test user’s password may have just been changed!

Rule of E2E Test Automation: a single test failure means the whole run is failed ( marked as red, started with JUnit).

Now think about a large suite…

For the challenge of managing test execution conflicts, check out this great Facebook presentation: “Continuous Integration at Facebook”.

Parallel Execution in the real world

I am a pioneer of parallel execution. After overcoming the hurdle of “Hard to maintain” (solution: Maintainable Automated Test Design & Functional Test Refactoring using Testing IDE) in 2008,

Test Automation Camel

I faced the second: long feedback time.

Test Automation Camel

At that time, there was hardly any mentioning the second challenge on the Internet, let alone the solution. Eventually, I solved this challenge by implementing parallel execution in CruiseControl, the granddaddy of CI servers. Later, I created the international award-winning BuildWise CT server. For more, check out My Innovative Solution to Continuous Testing: Parallel Automated End-to-End Test Execution.

This article is very long already; I will just highlight key points in Parallel Execution for real E2E Test Automation.

  • Use real machines as separate build agents. It is fairly cheap & easy to set up a reasonable-sized testing lab. Like Facebook, I used five MacMinis (far fewer than Facebook’s, thousands of), which is enough for my testing needs.
A recent run of 570 End-to-End Selenium tests in BuildWise CT server, with 5 build agents. I saved 75%, better than the 48% of the Playwright demo I did. Different from the demo, this is real for a large suite.
  • Leave Parallel Execution to a Continuous Testing Server. It is quite logical. In fact, traditional CI servers, Selenium Grid, and even Cypress Cloud follow a similar path. An automation framework just needs to ensure test execution is accurate, reliable, fully featured, and reasonably fast in sequential mode. A good CT server offers many great CT features that go beyond the framework. So, leave parallel execution to the CT Server. A solid proof is CT (with parallel execution and other features) works well with different automation and syntax frameworks. I have implemented successful E2E Test Automation with mixed frameworks, such as Watir, Selenium, Appium, Playwright, RSpec, Pytest, and Mocha.
A list of test automation project options supported by BuildWise CT Server

For more on parallel execution, check out my book: “Practical Continuous Testing: make Agile/DevOps real”.

Test Automation
Selenium
DevOps
Ci Cd Pipeline
Playwrights
Recommended from ReadMedium