Running Playwright tests in Azure DevOps: A step-by-step guide

Automating testing with Playwright in Azure DevOps improves efficiency and cost-effectiveness. It eliminates flaky tests, saving time and resources. In this blogpost Test Automation Engineer Kateryna Podopryhora explains how to seamlessly integrate Playwright into the Azure DevOps pipeline.

Continuing our CI/CD engineering series we dive into Playwright and testing in Azure DevOps. In the first blogpost Tom introduced composability for pipeline management, while Pascal highlighted the Backstage developer portal's templates for accelerated integration in the second post. Join me as we explore efficient testing practices in our Azure DevOps pipeline.

Streamlining test automation with Playwright in Azure DevOps

As a Platform Test Engineer, I was involved in the transition to Azure DevOps and the development of our new CI/CD engineering setup at funda. One of my primary responsibilities was implementing the test stages, encompassing unit, integration, and User Interface (UI) testing.

To accomplish effective UI testing, we adopted Playwright, an exceptional tool known for its reliable and efficient cross-browser web automation capabilities.

If you are eager to learn why we chose Playwright over Selenium and our experience with it, check out the enlightening article by Jome, titled 'Game changer: why we Ditched Selenium for Playwright.'

By incorporating Playwright into our testing process, we have obtained numerous advantages. One notable benefit has been the simplification of the setup process, allowing for smoother and quicker test implementation. Additionally, Playwright has helped us overcome the challenges of flaky tests, leading to significant time and resource savings. The integration of Playwright tests into our Azure DevOps pipeline offers a streamlined and efficient approach to test automation, aligning perfectly with our objectives.

In this guide, I'll take you step by step through the process of creating a generalized Playwright stage and seamlessly integrating it with Azure DevOps, providing test results within the interface. Along the way, I'll discuss the challenges we faced.

Examining structure of Playwright tests project

Let's begin by examining the structure of the Playwright tests project:

Docker configuration

To ensure consistency and productivity, we opted to run Playwright tests in a Docker container. This decision proved to be a game-changer, eliminating errors and inconsistencies caused by variations in development environments. The Docker file configuration simplifies the process:

Our package.json file contains the scripts required to run Playwright tests:

Taking testing to the next level with docker-compose

To further enhance our testing process, we implemented docker-compose, allowing us to run the subject under test (sut) and the tests in Azure DevOps and locally.

This approach simplifies debugging, saves time and resources, and means tests are run in a controlled environment. Furthermore, we set up our tests to run at build time, catching any issues early in the development cycle and reducing the risk of delays or setbacks.

The solution worked perfectly for us, except when we needed to rerun the same job in the pipeline. We encountered a failure in the task responsible for publishing test results, and after investigating, we discovered that the test results directory lacked the necessary permissions and couldn't be deleted.

As a result, the agent couldn't execute another Playwright stage successfully. To address this issue, we implemented a solution where the test results were stored in a temporary directory with a random name.

To gain a better understanding of these templates, please refer to the blogpost by Tom: How we use modular pipelines to empower our development teams

Composable template for Playwright test stage

Now, let's dive into the details of our composable template for the Playwright tests stage. This template streamlines the configuration process and ensures consistent and reliable execution of our tests.

First and foremost, the test container relies on the app container, establishing a dependency between the two using the 'dependsOn' parameter. This ensures that the test environment is correctly set up and ready to run the Playwright tests.
To ensure seamless execution, we provide npm credentials in an .npmrc file. This allows us to access any necessary dependencies and packages required for our tests.

Next, we log in to the Azure Container Registry (ACR) to pull the application's image. By authenticating with ACR, we ensure that we are working with the correct and up-to-date version of the application.

During the test execution, we create a temporary folder dedicated to storing the test results.

To execute the Playwright tests, we use docker-compose.

To make the test results easily accessible within the Azure DevOps interface, we generate a build artifact. This enables us to conveniently view and analyze the test results directly from the Azure DevOps dashboard.

Test Results Analysis in the Azure DevOps Dashboard

Once the Playwright tests are completed, reviewing the test results is very easy within the Azure DevOps dashboard. The Playwright test results are readily available, allowing for efficient analysis and troubleshooting. Screenshots and traces of failing tests are available for download, providing essential visual evidence and detailed information about any encountered issues.

To locate the Playwright test artifacts, you simply navigate to the output of the PlaywrightTests job. From there, you can effortlessly download the screenshots and traces associated with the failing tests.

Examining the azure-pipelines-build.yaml file

Lastly, let's take a look at our azure-pipelines-build.yaml file:

A testament to success

In conclusion, integrating Playwright tests into our CI/CD pipeline at funda has been a big success. As a Test Automation Engineer on the Platform team, I can confidently say that our decision to run Playwright tests in a Docker container has been a game-changer.

By leveraging Docker, we have achieved a high level of consistency and productivity in our testing process. Running Playwright tests within the container has eliminated errors and inconsistencies that could arise from variations in development environments. The Docker file configuration has streamlined the setup process, making it simpler and more efficient.

Overall, the integration of Playwright and Docker has provided us with a reliable and reproducible testing environment, both locally and in our CI/CD pipeline. This has not only simplified debugging but also saved valuable time and resources.

The success of our Playwright integration is a testament to the power and effectiveness of combining cutting-edge tools like Playwright with modern DevOps practices. We are confident that this approach will continue to drive our testing efforts and ensure the delivery of high-quality software products.

See also previous articles from the Platform Team:
Blogpost 1: How we use modular pipelines to empower our development teams
Blogpost 2: How to accelerate CI/CD integration with Backstage software templates