avatarZhimin Zhan

Summary

The author critically examines the claim that Playwright is superior to Selenium in terms of modernity and speed due to its clean architecture.

Abstract

The author critically examines the claim made in a YouTube video that Playwright is a superior web test automation framework compared to Selenium. The author argues that the claim that Playwright is modern and faster than Selenium due to its clean architecture is wrong. The author provides evidence to support this argument, including the fact that new does not necessarily mean better, and that software testers may not care about the architecture underneath. The author also points out that while Playwright may be faster in some cases, this has not been proven with a solid implementation. The author concludes by advising testers to focus on the job at hand rather than hypes.

Bullet points

  • The author critically examines the claim that Playwright is superior to Selenium in terms of modernity and speed due to its clean architecture.
  • The author argues that new does not necessarily mean better, and that software testers may not care about the architecture underneath.
  • The author points out that while Playwright may be faster in some cases, this has not been proven with a solid implementation.
  • The author advises testers to focus on the job at hand rather than hypes.

Correct Wrong “Playwright’s Advantage over Selenium” Part 1: “Playwright is Modern and Faster than Selenium”

New ≠ Modern ≠ Better. Faster? I haven’t seen a truly successful Playwright implementation with a large suite yet.

Recently, I watched a YouTube video, “Playwright vs Selenium: What Advantages Make Playwright the Winner in Automation Testing Battle 🏆”. Each of the presenter’s purported “Playwright Advantages” is either wrong or irrelevant in the context of real web test automation. I will elaborate on each of these points in this article series:

  • Playwright is Modern and Faster than Selenium 😒
  • Playwright has Parallel Execution Support” 👎🏽 (coming soon)
  • “Playwright has Native Auto-Waiting Mechanism” 👎🏽
  • “Playwright has a native test runner” 👎🏽
  • Playwright has native HTML Reporters” 👎🏽
  • Playwright Features Can be Configured in one Configuration” 👎🏽
  • Playwright supports a range of Testing Types, e.g. API Testing, Component Testing, …” 👎🏽
  • Playwright ARIA locator support” 👎🏽
  • Playwright UI Mode, CodeGen, Debugging support.” 👎🏽

How could this youtuber got it all wrong? Yes, it is very possible, because real automated engineers extremely rare. (Patrick Copeland, Google Senior Engineering Director, referred them as “Gold”)

Speaking of being off the mark totally, China International Capital Corporation made 10 predictions for the Chinese economy in 2023, and they all turned out to be incorrect!, that’s really hard, still, some ‘financial experts’ managed to accomplish that.

Table of Contents: · My Objectiveness on Playwright1. I do know a bit about Playwright2. I added Playwright support in my testing IDE (TestWise) and award-winning CT server (BuildWise)3. Selenium WebDriver is Excellent4. Playwright is winning over Cypress, but not Selenium WebDriver. · Claim 1, “Playwright is Modern and Faster than Selenium, due to its Clean Architecture“. Wrong!1. New ≠ Modern ≠ Better.2. “Clean Architecture”? Do we testers really care?3. “Playwright faster”, maybe. But I haven’t yet seen a solid Playwright implementation to support that.

My Objectiveness on Playwright

The first question in Playwright fans’ minds is, “Zhimin, you are being subjective”. How many statements on the Internet are not subjective, including the “Playwright is Modern and Faster than Selenium”, then?

Unlike many others, I will provide objective proof to support my statements.

1. I do know a bit about Playwright

Many readers know I mostly do web test automation using raw Selenium WebDriver + RSpec (since 2011). I have used and refine other frameworks as well, including Playwright. My daughter used Playwright for her intern job, where she crushed the so-called ‘best senior test automation engineer’ there (an interesting story).

Also, I have rescued failed Playwright test automation (upgrading to raw Selenium WebDriver).

2. I added Playwright support in my testing IDE (TestWise) and award-winning CT server (BuildWise)

I am an old-school engineer. If necessary, I would dive deep (really deep) for new hyped technology before commenting on it. In this case, I added Playwright support to my tools: TestWise and BuildWise. This also shows that I am not totally against Playwright; it is just not as good as raw Selenium WebDriver.

Check out the following articles (by my daughter).

3. Selenium WebDriver is Excellent

This is a comparison article, so I must mention my Selenium experience for readers who don’t know me well.

Below is a case study of my WhenWise app test suite, consisting of 570 user story-level End-to-End UI tests in raw Selenium WebDriver + RSpec.

This is just one of my Selenium suites that enabled me to do “Daily Production Releases” for all my web apps since 2012.

4. Playwright is winning over Cypress, but not Selenium WebDriver.

There is a clear trend that Playwright is taking over Cypress (my prediction in this article, 5 months ago. One tech blogger disagreed, I used the reference he mentioned to prove him wrong). The most recent two independent surveys yielded a similar result:

https://zhiminzhan.medium.com/selenium-webdriver-is-still-the-best-web-test-automation-framework-in-2024-3c0c68d38965

The “old” Selenium WebDriver leads by a huge margin.

Claim 1, “Playwright is Modern and Faster than Selenium, due to its Clean Architecture“. Wrong!

The claim (screenshotted) in that YouTube video.

1. New ≠ Modern ≠ Better.

Communists claimed their system was “Newer” and “Modern” too.

US President Biden called Chinese Chairman Xi Jinping “a dicator”, who changed the consititution to have unlimited turns. The slogan Xi used: “Chinese Dream”, “New Age”. He even invented a new terminology, “Whole-process people’s democracy”, claiming this new process (no rival parties, people no voting at all, justice system has to serve the chinese communist party) is “better” than the western democracy.

Playwright is newer, but it does not necessarily mean “Modern” or “Better”. Several of the following- already-failed automation frameworks could/did claim that, too.

Too Many Failed JavaScript Test Automation Frameworks!. Each of them is newer than Selenium.

2. “Clean Architecture”? Do we testers really care?

By the way, the author also got the architecture name wrong.

To my knowledge, there is no “Chrome Driver Protocol” 😱. He might mean “Chrome DevTools Protocol” (CDP). If so, Selenium v4 supports it too.

In terms of architecture, Selenium WebDriver clearly wins. Why? It conforms with the WebDriver standard, designed and refined by top experts for years. W3C sets the standards for all web technologies, and all browser vendors support WebDriver. How can any other top that?

Do software testers really need to care about the architecture underneath? I don’t think so. I consider myself a highly technical, hands-on test automation engineer (100X and international award-winning programmer) and have reviewed dozens of test automation frameworks/tools. When working as an automation tester at jobs, the thought of a test automation framework architecture never crossed my mind.

For example, to click a link using Selenium WebDriver,

driver.find_element(:link, "TestWise").click

Do I need to care about the JSON wired protocol WebDriver uses (underneath)? When authoring books on Selenium WebDriver, I did, but not for scripting steps.

For web test automation, knowing a framework conforms to W3C standards and major browser vendors support it is good enough for me.

“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released.” — DAMIEN SERENI, Engineering Director at Facebook, at Selenium 2013 conference.

My advice to testers: focus on the job at hand, not hypes.

3. “Playwright faster”, maybe. But I haven’t yet seen a solid Playwright implementation to support that.

Running a handful of tests might show Playwright slightly faster; the difference is minor, though (as revealed by several benchmarks, don’t add parallel execution, which I will explain in a separate article). Remember, Cypress claimed that, too.

https://zhiminzhan.medium.com/comments-on-the-claims-of-7-ways-cypress-is-different-all-false-wrong-or-lie-74648dfd1932#43bf

The Cypress claim provided no proof (test suite and timing). Various benchmark tests, like this one, showed Cypress is the slowest among Selenium, Cypress and Playwright. So “much, much faster” is a joke.

For individual End-to-end test case execution, nearly ALL web automation frameworks are FAST enough. After all, the majority of E2E test execution time is on the app itself.

Please note that a fast execution on a few sample tests (which I refer to as demo mode) does not mean it will still be fast for executing a large suite. Also, don’t confuse Playwright parallel mode here (which will be covered in a later dedicated article).

“Because of what it (CDP) was designed to do, it is a chatty protocol, and not a user-facing API that allows you to introspect into the browser. Worse, introduce a network hop between the test and the browser, this chattiness leads to slower tests as the network latency increases.” — Simon Stewart, 2020–11–06

We care about the execution speed of a large suite, say 200 End-to-End User Story Level tests. However, I have never witnessed a single case of OK-level implementation of Cypress or Playwright test automation. My definition of OK level (2nd of AgileWay Continuous Testing Grading): 50 User story level UI tests that run daily and, on most days, remain valid. That’s not a big ask, right? I haven’t seen one using Cypress or Playwright. The successful ones were all using raw Selenium WebDriver.

In terms of speed, check out my article, “Playwright vs Selenium Speed Comparison”. For individual test execution, the difference is minor (here is another benchmark). For real web test automation, with a large test suite, you need to run them in a good CT server on a parallel testing lab.

Related reading:

Test Automation
Selenium
Selenium Webdriver
Playwrights
Software Testing
Recommended from ReadMedium