How I ran over 200 e2e tests in the pipeline

End-to-End (E2E) testing is a crucial step in ensuring the robustness and reliability of your application. In this guide, we’ll explore how I successfully ran over 200 E2E tests in my pipeline by leveraging Nx modularization, GitLab change detection, or even a custom test runner. This approach not only saved time but also provided a more focused and efficient testing process.

Jump to the solution 🔥

Nx affected and Gitlab change detection

While Nx modularization and GitLab change detection provide a solid foundation for E2E testing scalability, an alternative route involves placing a stronger emphasis on a custom Cypress test runner. This approach allows for even more targeted testing, ensuring that only the necessary Cypress tests are executed in response to changes.

Nx affected

Nx modularization remains a fundamental component of this alternative route, providing a structured and organized workspace. By breaking down the project into smaller, manageable applications and libraries, you gain the advantage of selective testing for improved maintenance.

my-app/
|-- apps/
|   |-- app1/
|   |-- app2/
|-- libs/
|   |-- shared/
|   |-- feature1/
|   |-- feature2/

once you got yourself organized repository, as mentioned in NX docs, you can set package.json file, ..file needed for every nx module… in this file, you can setup targets specific for the module

{
  "name": "myApp",
  "$schema": "../../node_modules/nx/schemas/project-schema.json",
  "sourceRoot": "apps/myApp/src",
  "projectType": "application",
  "tags": ["scope:app", "type:app"],

  "targets": {
     "e2e": {
      "executor": "nx:run-commands",
      "options": {
        "command": "nx run myApp:e2e --browser=chrome"
      }
    },
  }
}

After that, you just run nx affected — target=e2e are you're all set and done!

Gitlab change detection

GitLab change detection continues to play a crucial role in this alternative approach. Leveraging GitLab CI configuration allows you to set up jobs that respond specifically to changes in designated directories or files. This feature minimizes unnecessary test runs, enhancing the efficiency of your E2E testing pipeline.

stages:
  - test

e2e_tests:
  stage: test
  script:
    - run_cypress_tests.sh
  only:
    changes:
      - 'apps/**/*'

Custom e2e test runner 🚀

The focal point of this route is the development of a custom Cypress test runner. This runner goes beyond detecting basic changes and focuses on the modification, removal, or movement of the data-testId property within Cypress tests. This nuanced approach ensures that only relevant tests are executed, significantly reducing testing time and resources.

Firstly, we create a script that detects changed modules

1. Detects changes in files related to the data-testId attribute.
2. Extracts the module from each changed file path.
3. Runs Cypress tests for each affected module.

const { execSync } = require('child_process');
const fs = require('fs');

function getChangedFilesWithTestId(diffOutput) {
    const filesWithTestIdChanges = [];

    const fileLines = diffOutput.split('\n');
    let currentFile = null;
    let testIdChange = false;

    fileLines.forEach(line => {
        if (line.startsWith('diff --git')) {
            // Extract file path from the diff header
            currentFile = line.split(' b/')[1];
            testIdChange = false;
        } else if (line.startsWith('+') && line.includes('data-testId')) {
            testIdChange = true;
        } else if (line.startsWith('-') && line.includes('data-testId')) {
            testIdChange = true;
        } else if (line.startsWith('@@') && testIdChange) {
            // If 'data-testId' change detected, add the file to the list
            if (currentFile) {
                filesWithTestIdChanges.push(currentFile);
            }
            currentFile = null;
            testIdChange = false;
        }
    });

    return filesWithTestIdChanges;
}

// Run 'git diff' command to get the changes
const gitDiffOutput = execSync('git diff master').toString();

// Get files with 'data-testId' changes
const filesWithTestIdChanges = getChangedFilesWithTestId(gitDiffOutput);

// Here you have the list of files with changes in the 'data-testId' attribute
console.log('Files with data-testId changes:', filesWithTestIdChanges);

after that we can just iterate though the output of this function, to run tests associated with my changes

function runCypressTestsForModule(moduleName) {
    // Execute Cypress tests for the specified module
    execSync(`cypress run --spec cypress/**/${moduleName}/**/*.spec.js`);
}

In a monorepo where shared modules are integral to the overall architecture, preventing every change in shared modules from triggering the execution of every single test is crucial for optimizing CI/CD pipelines. While the traditional approach using NX’s nx affected commands might lead to running all tests associated with shared modules, the custom Node.js script offers a more targeted and efficient solution.

While this approach offers a targeted and efficient strategy for Cypress test execution, it requires regular maintenance and customization to adapt to specific project requirements. Users should carefully validate and enhance the script based on the evolving needs of their development workflows and codebase structures.

Estimating Time Savings 🕐

The actual time savings achieved by implementing this script can vary based on factors such as the frequency of code changes, the size of the codebase, and the nature of the tests. However, it is reasonable to expect substantial time savings, potentially reducing CI/CD pipeline durations by a significant percentage. In scenarios with frequent changes and a sizable test suite, the script’s impact on CI time can be particularly pronounced, leading to a more streamlined and responsive development workflow.