Summary

The article critically examines the claim that Playwright's native test runner is an advantage over Selenium, arguing instead that it introduces unnecessary complexity and challenges.

Abstract

The article is the fourth part in a series that debunks misconceptions about Playwright's advantages over Selenium. It focuses on the claim that Playwright's native test runner is a benefit, contending that this feature is more of a disadvantage. The author asserts that by convention, an automation framework should not include a test runner, as this responsibility belongs to test syntax frameworks like RSpec or Mocha. The article points out that Playwright's approach, which bundles the test runner within the automation framework, leads to complications such as the need for two separate packages (playwright and @playwright/test). The author expresses a preference for the decoupling of automation and test frameworks, which allows for more flexibility and the use of established test runners that are already integrated with CI/CT servers. The article also lists several specific issues with Playwright's test runner, including non-intuitive syntax, inability to run tests from any folder, default headless mode, and bugginess, suggesting that these complexities are unnecessary and do not align with the author's experience in maintaining large test suites.

Opinions

The author believes that Playwright's native test runner is not an advantage but a drawback, as it goes against the conventional separation of automation frameworks and test syntax frameworks.
It is argued that the Playwright test runner adds unnecessary complexity, such as requiring two separate packages and dealing with package names containing the '@' symbol.
The author prefers using established test syntax frameworks like RSpec or Mocha, which are mature and widely supported in the industry.
The article suggests that Playwright's test runner complicates the integration with CI/CT servers and hinders the ability to leverage previous knowledge or custom scripts.
Specific criticisms of the Playwright test runner include its non-intuitive command syntax, the inability to run tests from any folder, the default headless mode which the author finds less effective for catching defects, and the individual test case execution syntax that is prone to changes in test case names.
The author reports finding bugs in the Playwright test runner, which raises concerns about its reliability compared to more mature test frameworks.
The preference for running tests in headed mode by default is emphasized, as it allows for the casual detection of issues in the application under test.

Correcting Wrong “Playwright’s Advantage over Selenium” Part 4: “Playwright has a native test runner”

It is more a Con, not a Pro.

Continue to correct the fourth wrong claim in this YouTube video, “Playwright vs Selenium: What Advantages Make Playwright the Winner in Automation Testing Battle 🏆”.

This article series:

“Playwright is Modern and Faster than Selenium” 😒
“Playwright has Parallel Execution Support” 👎🏽
“Playwright has Native Auto-Waiting Mechanism” 👎🏽
“Playwright has a native test runner” 👎🏽
“Playwright has native HTML Reporters” 👎🏽 (coming soon)
“Playwright Features Can be Configured in one Configuration” 👎🏽
“Playwright supports a range of Testing Types, e.g. API Testing, Component Testing, …” 👎🏽
“Playwright ARIA locator support” 👎🏽
“Playwright UI Mode, CodeGen, Debugging support.” 👎🏽

Table of Contents: · Claim 4: “Playwright has native playwright tests” · I don’t like “npx playwright test” runner · There are good reasons why most automation frameworks do not provide their runners. ∘ Why should Automation and Test Syntax Frameworks remain loosely coupled? · The problems with the Playwright Runner.

Claim 4: “Playwright has native playwright tests”

Selenium WebDriver is an automation framework, and so are Protractor and Playwright. By convention, a web automation framework drives the app in the browser, and should not have its test runner. The test syntax, runner and report are the responsibility of Test Syntax Framworks, such as RSpec, JUnit, PyTest, Mocha and Cucumber.

Test Automation Framework = Automation Framework + Test Syntax Framework

That’s why we hear “Watir + RSpec”, “Protractor + Mocha”, “Selenium + RSpec”, “Appium + RSpec”, or “Selenium + Cucumber”. As a comparison, back in the old days, commercial automation tools such as QTP intentionally blurred the boundary between automation and test syntax frameworks.

For readers who are not familiar with RSpec, the first BDD framework (first release in 2007). BTW, BDD framework ≠ Gherkin. RSpec v3.8 has 210 million downloads. Mocha and Pytest are considered clones of RSpec; they work in a similar way. I think RSpec is still the best test syntax framework for E2E test automation.

While combining automation and test syntax framework (including test runner) is not strictly forbidden, it is more a ‘Con’ than a ‘Pro’ (I will explain shortly).

In Playwright's case, it is more tricky. Because of this runner, we have two packages playwright and @playwright/test (a package contains a test syntax framework with runner). A package name containing ‘@’ symbol! Don’t you think this adds extra complexity unnecessarily?

For example, the description for those two are exactly the same.

I don’t like “npx playwright test” runner

Frankly, I don’t like the “npx playwright test” runner syntax.

Mocha is the widely used test syntax framework in the JS community. You can certainly achieve E2E test automation goals with Playwright + Mocha.

Some Playwright fans may can’t wait to argue, “There are benefits of Playwright test runner, such as … ”, which I disagree with; See my explanation later. As a test automation engineer, I verify the results. I haven’t yet seen one successful Playwright E2E test automation with 50+ user-story level UI tests running valid as daily regression testing. But I do know raw Selenium WebDriver + RSpec works very well, like the one below.

Showcase a 500+ End-to-End (via UI) Test Suite: E2E Test Automation is Surely Feasible for…

Implementing End-to-End Test Automation, the most valuable type of testing, is most QA Engineers’ responsibility. Be a…

zhiminzhan.medium.com

There are good reasons why most automation frameworks do not provide their runners.

Before Playwright, the convention was that an open-source automation framework is loosely coupled with test frameworks, such as Selenium (with multiple test frameworks, e.g. RSpec, Junit, Pytest, Mocha, …) and even Cypress (default Mocha). In other words, automation framework creators don’t care how users run it, leaving the choice to the users (to choose a test syntax framework).

Playwright with Mocha against Chrome browser

Write Playwright tests in Mocha to take advantage of its wide support, such as test execution in CI/CT servers. By the…

zhiminzhan.medium.com

Why should Automation and Test Syntax Frameworks remain loosely coupled?

Test syntax runners have been quite mature for over two decades. There is not much to it. For example, JUnit still dominates unit testing in Java. There is no need to create another unit testing library in Java.
Test Automation Engineers are unable to leverage their previous knowledge or custom scripts. Imagine a scenario where a company migrate its Cypress test suite to Playwright. Initially, the automated testers believed it was merely a change in the automation framework, only to discover later that Mocha was also replaced in the process.
This introduces extra work to integrate with CI/CT servers. The DevOps engineers are already familiar with the common Test syntax runners, such as JUnit, RSpec, Pytest and Mocha. Introducing the npx playwright test is extra work.

The problems with the Playwright Runner.

Long-time readers know I have been maintaining several large-size E2E (via UI) test suites. Seasoned software engineers understand that the challenge of maintaining a 200-test suite is NOT just twice as difficult as a 100-test suite; it’s more than 10 times harder. I can affirm that none of “those cool features” in Playwright, including the test runner, resemble the useful ones I use in my daily maintenance of large test suites.

Instead, I found Playwright Test Runner awkward.

npx playwright test syntax is not intuitive. It was the first time I saw a “three-word” test execution command.
You cannot run the test from any folder For example, I can run rspec /Users/courtney/demo/login_spec.rb from anywhere as long as I provide the target test script file path. With Playwright Runner, I got an error like this, “Error: Playwright Test did not expect test.describe.configure() to be called here.”
I prefer running tests in headed mode by default. The default mode of Playwright Runner is headless ( add — -headed for for normal mode), which is not good. By executing tests in normal mode, I have been picking up many defects/issues in the app by causal looking). The tiny percentage of speed gain in headless is not worth it. After all, the majority of individual test automation script effort (create, refine, debug) is in the normal mode anyway. For more, check out my articles: Headless Browser Testing Clarified and Why Fake Automated Testers and Software Test Architects Love ‘Headless Browser Testing’?
I don’t like its individual test case execution syntax: `npx playwright test -g “add a todo item”` A simple reason is that the test case name is often updated. The RSpec’s by a line number way is better.
Buggy I found a different behaviour (more like a bug) when running on Windows and macOS. It is related to / separator. Yes, it is a minor issue, but common test syntax frameworks have matured for two decades. This made me feel frustrated and worried about more bugs down the stream.

Anyway, it is all the complexity that is not necessary to me. (Check out my article, Software Complexity Assessment)

Some Playwright fans might debate, “The reports from the test runner are good”. I disagree. This would be the topic for the next article.

Related reading: