Automating visual UI tests with Playwright and GitHub Actions

Visual UI testing (also known as regression testing) is a testing technique to validate that your changes don't have any unexpected impact on the UI. Typically, such tests take an image snapshot of the entire application under test or a specific element and then compare the image to a previously approved baseline image. If the images are the same (within a set pixel tolerance), it is determined that the web application looks the same to the user. If there are differences, then there has been some change to the DOM layout, fonts, colors, or other visual properties that need to be investigated.

This post will explore how to automate visual regression tests of a "modern" web application using Playwright and GitHub actions. The goal is to build a testing setup that checks for UI regressions on each pull request and allows selectively updating the baseline images when needed.

Basic knowledge of JavaScript (or TypeScript) and GitHub Actions is recommended.

We assume you'll integrate this setup into an existing web application. If you want to try it from scratch, I recommend scaffolding a new web application using Vite.

You can find a complete example of the resulting application in this GitHub repo.

For reference, here's what the tiny web application we're testing looks like:

Playwright setup

For our testing setup, we'll use Playwright, an E2E testing framework. I like Playwright because it provides a great developer experience and nice defaults out of the box, but you can achieve the same result with similar tools such as Cypress or Selenium.

To install Playwright cd into our project and run the following command:

npm init playwright

We'll be prompted with a few different options (e.g. JavaScript vs TypeScript, etc.). When asked, we should add a GitHub Actions workflow to run our tests on CI easily:

✔ Add a GitHub Actions workflow? (y/N) · true

✔ Add a GitHub Actions workflow? (y/N) · true

Once done, Playwright will add an example test in ./tests/example.spec.ts, an example E2E file in ./tests-examples/demo-todo-app.spec.ts (that we can ignore), and the Playwright configuration file in ./playwright.config.ts.

✔ Success! Created a Playwright Test project at ~/your-project-dir

Inside that directory, you can run several commands:

  npx playwright test
    Runs the end-to-end tests.

  npx playwright test --project=chromium
    Runs the tests only on Desktop Chrome.

  npx playwright test example
    Runs the tests in a specific file.

  npx playwright test --debug
    Runs the tests in debug mode.

  npx playwright codegen
    Auto generate tests with Codegen.

We suggest that you begin by typing:

    npx playwright test

And check out the following files:
  - ./tests/example.spec.ts - Example end-to-end test
  - ./tests-examples/demo-todo-app.spec.ts - Demo Todo App end-to-end tests
  - ./playwright.config.ts - Playwright Test configuration

Visit https://playwright.dev/docs/intro for more information. ✨

✔ Success! Created a Playwright Test project at ~/your-project-dir

Inside that directory, you can run several commands:

  npx playwright test
    Runs the end-to-end tests.

  npx playwright test --project=chromium
    Runs the tests only on Desktop Chrome.

  npx playwright test example
    Runs the tests in a specific file.

  npx playwright test --debug
    Runs the tests in debug mode.

  npx playwright codegen
    Auto generate tests with Codegen.

We suggest that you begin by typing:

    npx playwright test

And check out the following files:
  - ./tests/example.spec.ts - Example end-to-end test
  - ./tests-examples/demo-todo-app.spec.ts - Demo Todo App end-to-end tests
  - ./playwright.config.ts - Playwright Test configuration

Visit https://playwright.dev/docs/intro for more information. ✨

The Playwright configuration file generated by the scaffolding process (./playwright.config.ts) provides some nice defaults, but I recommend applying a couple of changes.

First, to simplify our setup, update the projects list to run our tests only on Chromium:

playwright.config.ts

/* Configure projects for major browsers */
projects: [
  {
    name: "chromium",
    use: {
      ...devices["Desktop Chrome"],
    },
  },

playwright.config.ts

/* Configure projects for major browsers */
projects: [
  {
    name: "chromium",
    use: {
      ...devices["Desktop Chrome"],
    },
  },

We can update the project list later on, to include more browsers/devices, if needed.

Then, configure the webServer section so that Playwright can automatically serve our application when we run our tests:

playwright.config.ts

/* Run your local server before starting the tests */
webServer: {
  command: 'npm run dev --port 8080',
  port: 8080,
  reuseExistingServer: true
},

playwright.config.ts

/* Run your local server before starting the tests */
webServer: {
  command: 'npm run dev --port 8080',
  port: 8080,
  reuseExistingServer: true
},

Generate the initial snapshots

The example test generated by Playwright (./tests/example.spec.ts) is not a visual regression test, so let's delete its content and add a tiny test that suits our needs:

tests/example.spec.ts

import { test, expect } from "@playwright/test";
 
test("example test", async ({ page }) => {
  await page.goto("/"); // The baseURL here is the webServer URL
  await expect(page).toHaveScreenshot();
});

tests/example.spec.ts

import { test, expect } from "@playwright/test";
 
test("example test", async ({ page }) => {
  await page.goto("/"); // The baseURL here is the webServer URL
  await expect(page).toHaveScreenshot();
});

For visual regression testing, Playwright includes the ability to produce and visually compare snapshots using await expect(page).toHaveScreenshot(). On first execution, Playwright test will generate reference snapshots. Subsequent runs will compare against the reference.

We're finally ready to run your test:

npx playwright test

npx playwright test

The test runner will say something along the line of:

Error: example.spec.ts-snapshots/example-test-1-chromium-darwin.png is missing in snapshots, writing actual.

Error: example.spec.ts-snapshots/example-test-1-chromium-darwin.png is missing in snapshots, writing actual.

That's because there was no baseline image yet, so this method took a bunch of snapshots until two consecutive snapshots matched, and saved the last snapshot to file system. It is now ready to be added to the repository (in tests/example.spec.ts-snapshots/example-test-1-chromium-darwin.png):

Now, if we run our test again:

npx playwright test

npx playwright test

The test should now succeed because the current UI matches the UI of the reference snapshot generated in the previous run.

Update the snapshots locally

Let's move to the interesting part.

If we make any visual change to the UI and rerun our tests, they will fail, and Playwright will show us a nice diff between the "actual" and "expected" snapshot:

In cases such as this one, when we want to make some voluntary changes to a page, we need to update the reference snapshots. We can do this with the --update-snapshots flag:

npx playwright test --update-snapshots

npx playwright test --update-snapshots

[chromium] › example.spec.ts:3:1 › example test
tests/example.spec.ts-snapshots/example-test-1-chromium-darwin.png is re-generated, writing actual.

[chromium] › example.spec.ts:3:1 › example test
tests/example.spec.ts-snapshots/example-test-1-chromium-darwin.png is re-generated, writing actual.

Now that we've seen how to run visual tests and update the snapshots locally, we're ready to move to the CI flow.

Running the tests in CI using GitHub Actions

GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that allows you to automate your build, test, and deployment pipeline. With GitHub Actions we can create workflows that build and test every pull request to a repository.

A good starting point for a visual regression testing setup is to run the tests on each pull request creation and update. Luckily, Playwright already generated a handy GitHub Actions workflow to run tests for this specific use case in .github/workflows/playwright.yml.
This workflow should work perfectly fine out-of-the-box, but I recommend tweaking it a bit to:

Install only the browsers we're targeting with our tests (--with deps chromium);
Save the test result artifacts only when tests fail to avoid storing unnecessary artifacts (if: failure()).

.github/workflows/playwright.yml

 name: Playwright Tests
 
 on:
   push:
     branches: [main, master]
   pull_request:
     branches: [main, master]
 
 jobs:
   test:
     timeout-minutes: 60
     runs-on: ubuntu-latest
     steps:
       # Checkout and setup
       - uses: actions/checkout@v2
       - uses: actions/setup-node@v2
         with:
           node-version: "14.x"
       - name: Install dependencies
         run: npm install
       - name: Install Playwright Browsers
-        run: npx playwright install --with-deps
+        run: npx playwright install --with-deps chromium
       # Run Playwright tests
       - name: Run Playwright tests
         run: npx playwright test
       # Upload the test result artifacts
       - uses: actions/upload-artifact@v2
-        if: always()
+        if: failure()
         with:
           name: playwright-report
           path: playwright-report/
           retention-days: 30

.github/workflows/playwright.yml

 name: Playwright Tests
 
 on:
   push:
     branches: [main, master]
   pull_request:
     branches: [main, master]
 
 jobs:
   test:
     timeout-minutes: 60
     runs-on: ubuntu-latest
     steps:
       # Checkout and setup
       - uses: actions/checkout@v2
       - uses: actions/setup-node@v2
         with:
           node-version: "14.x"
       - name: Install dependencies
         run: npm install
       - name: Install Playwright Browsers
-        run: npx playwright install --with-deps
+        run: npx playwright install --with-deps chromium
       # Run Playwright tests
       - name: Run Playwright tests
         run: npx playwright test
       # Upload the test result artifacts
       - uses: actions/upload-artifact@v2
-        if: always()
+        if: failure()
         with:
           name: playwright-report
           path: playwright-report/
           retention-days: 30

Once we commit this workflow in the GitHub repo, submitting a new pull request will trigger the Playwright tests.

However, our test will likely fail at this point because the reference snapshots in our repo have been captured in an environment that differs from the one used on CI (unless we created them on a machine running Ubuntu).

For more information on the test result, we can check either the GitHub Action output or the test artifacts:

For example, if we generated the reference snapshots on macOS, we'll receive an error stating that the snapshots for Linux haven't been found:

Error: tests/example.spec.ts-snapshots/example-test-1-chromium-linux.png is missing in snapshots

Error: tests/example.spec.ts-snapshots/example-test-1-chromium-linux.png is missing in snapshots

Let's see how we can update the reference snapshots for the CI environment.

Updating the snapshots in CI with a pull-request comment

Generating the reference snapshots locally using --update-snapshots was a piece of cake, but on CI, it's a different story because we must decide where, how, and when to store them.

There are plenty of ways we can handle this flow, but for the sake of simplicity, let's start simple.
One pattern that has worked well for me is to use a GitHub Action workflow to update the reference snapshots when a specific comment with a "/update-snapshots" text is posted in a pull request.
The idea is that whenever we submit a pull request that we expect to impact the UI, we can post the "/update-snapshots" comment so that CI will generate and commit the updated snapshots in the pull request branch.

.github/workflows/update-snapshots.yml

# This workflow's goal is forcing an update of the reference snapshots used
# by Playwright tests. It runs whenever you post a new pull request comment
# that strictly matches the "/update-snapshots".
# From a high-level perspective, it works like this:
# 1. Because of a GitHub Action limitation, this workflow is triggered on every
#    comment posted on a issue or pull request. We manually interrupt it unless
#    the comment content strictly matches "/update-snapshots" and we're in a
#    pull request.
# 2. Use the GitHub API to grab the information about the branch name and SHA of
#    the latest commit of the current pull request.
# 3. Update the Playwright reference snapshots based on the UI of this branch.
# 4. Commit the newly generated Playwright reference snapshots into this branch.
name: Update Snapshots
 
on:
  # It looks like you can't target PRs-only comments:
  # https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request_comment-use-issue_comment
  # So we must run this workflow every time a new comment is added to issues
  # and pull requests
  issue_comment:
    types: [created]
 
jobs:
  updatesnapshots:
    # Run this job only on comments of pull requests that strictly match
    # the "/update-snapshots" string
    if: ${{ github.event.issue.pull_request && github.event.comment.body == '/update-snapshots'}}
    timeout-minutes: 60
    runs-on: ubuntu-latest
    steps:
      # Checkout and do a deep fetch to load all commit IDs
      - uses: actions/checkout@v2
        with:
          fetch-depth: 0 # Load all commits
          token: ${{ secrets.GITHUB_TOKEN }}
      # Get the SHA and branch name of the comment's pull request
      # We must use the GitHub API to retrieve these information because they're
      # not accessibile within workflows triggered by "issue_comment"
      - name: Get SHA and branch name
        id: get-branch-and-sha
        run: |
          sha_and_branch=$(\
            curl \
              -H 'authorization: Bearer ${{ secrets.GITHUB_TOKEN }}' \
              https://api.github.com/repos/${{ github.repository }}/pulls/${{ github.event.issue.number }} \
            | jq -r '.head.sha," ",.head.ref');
          echo "::set-output name=sha::$(echo $sha_and_branch | cut -d " " -f 1)";
          echo "::set-output name=branch::$(echo $sha_and_branch | cut -d " " -f 2)"
      # Checkout the comment's branch
      - name: Fetch Branch
        run: git fetch
      - name: Checkout Branch
        run: git checkout ${{ steps.get-branch-and-sha.outputs.branch }}
      # Setup testing environment
      - uses: actions/setup-node@v2
        with:
          node-version: "14.x"
      - name: Install dependencies
        run: npm install
      - name: Install Playwright browsers
        run: npx playwright install --with-deps chromium
      # Update the snapshots based on the current UI
      - name: Update snapshots
        run: npx playwright test --update-snapshots --reporter=list
      # Commit the changes to the pull request branch
      - uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: "[CI] Update Snapshots"

.github/workflows/update-snapshots.yml

# This workflow's goal is forcing an update of the reference snapshots used
# by Playwright tests. It runs whenever you post a new pull request comment
# that strictly matches the "/update-snapshots".
# From a high-level perspective, it works like this:
# 1. Because of a GitHub Action limitation, this workflow is triggered on every
#    comment posted on a issue or pull request. We manually interrupt it unless
#    the comment content strictly matches "/update-snapshots" and we're in a
#    pull request.
# 2. Use the GitHub API to grab the information about the branch name and SHA of
#    the latest commit of the current pull request.
# 3. Update the Playwright reference snapshots based on the UI of this branch.
# 4. Commit the newly generated Playwright reference snapshots into this branch.
name: Update Snapshots
 
on:
  # It looks like you can't target PRs-only comments:
  # https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request_comment-use-issue_comment
  # So we must run this workflow every time a new comment is added to issues
  # and pull requests
  issue_comment:
    types: [created]
 
jobs:
  updatesnapshots:
    # Run this job only on comments of pull requests that strictly match
    # the "/update-snapshots" string
    if: ${{ github.event.issue.pull_request && github.event.comment.body == '/update-snapshots'}}
    timeout-minutes: 60
    runs-on: ubuntu-latest
    steps:
      # Checkout and do a deep fetch to load all commit IDs
      - uses: actions/checkout@v2
        with:
          fetch-depth: 0 # Load all commits
          token: ${{ secrets.GITHUB_TOKEN }}
      # Get the SHA and branch name of the comment's pull request
      # We must use the GitHub API to retrieve these information because they're
      # not accessibile within workflows triggered by "issue_comment"
      - name: Get SHA and branch name
        id: get-branch-and-sha
        run: |
          sha_and_branch=$(\
            curl \
              -H 'authorization: Bearer ${{ secrets.GITHUB_TOKEN }}' \
              https://api.github.com/repos/${{ github.repository }}/pulls/${{ github.event.issue.number }} \
            | jq -r '.head.sha," ",.head.ref');
          echo "::set-output name=sha::$(echo $sha_and_branch | cut -d " " -f 1)";
          echo "::set-output name=branch::$(echo $sha_and_branch | cut -d " " -f 2)"
      # Checkout the comment's branch
      - name: Fetch Branch
        run: git fetch
      - name: Checkout Branch
        run: git checkout ${{ steps.get-branch-and-sha.outputs.branch }}
      # Setup testing environment
      - uses: actions/setup-node@v2
        with:
          node-version: "14.x"
      - name: Install dependencies
        run: npm install
      - name: Install Playwright browsers
        run: npx playwright install --with-deps chromium
      # Update the snapshots based on the current UI
      - name: Update snapshots
        run: npx playwright test --update-snapshots --reporter=list
      # Commit the changes to the pull request branch
      - uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: "[CI] Update Snapshots"

If we publish this workflow, create a new pull request, and add a "/update-snapshots" comment, CI will take care of generating the reference snapshots.

Nice!

That's it — we have fully automated visual regression testing on our Continuous Integration. Whenever new changes are posted in new pull requests, our tests will ensure we're not mistakenly making changes to the UI. We can also update our baseline snapshots by posting a comment in the pull request.

If your project supports preview links (such as the ones automatically generated by Netlify, Vercel, etc.), keep on reading. Otherwise, jump to the Conclusion section below.

Run Playwright tests against deploy preview (Netlify, Vercel, etc.)

If your web application runs on platforms such as Netlify and Vercel, it's likely you're using their "Deploy Preview" feature: a way to preview pull request changes without impacting the web application in production. Deploy previews are enabled by default for GitHub pull requests, and they work by deploying them to a unique URL different from the one your production site uses.

If your web application uses deploy previews, we can integrate them into our visual regression testing workflows to compare the snapshots against them instead of the local web server.
This approach brings two benefits. First, we avoid spinning up a local webserver just for running the tests. Second, by running our tests against a preview link, we'll produce more reliable snapshots because preview links are a 1:1 representation of what we should see in production.

From a high-level view, there are three main changes we need to make in our codebase to run our Playwright tests against deploy previews:

Allow passing the URL to test as a parameter to make our tests aware of the deploy preview link.
Update our GitHub Action workflows to wait for the deploy preview to be completed.
Update our GitHub Action workflows to pass to Playwright the deploy preview URL (as an environment variable).

Here's how we can achieve this in Netlify (if you're using Vercel or any other platform that supports deploy preview, the changes should be almost identical).

First, update the use.baseURL value of playwright.config.ts to receive the deploy URL as an environment variable (WEBSITE_URL):

playwright.config.ts

use: {
  baseURL: process.env.WEBSITE_URL,
}

playwright.config.ts

use: {
  baseURL: process.env.WEBSITE_URL,
}

Also, let's disable Playwright's web server if a WEBSITE_URL environment variable is provided:

webServer: process.env.WEBSITE_URL
  ? undefined
  : {
    command: "npm run dev --port 8080",
    port: 8080,
    reuseExistingServer: true,
  },

webServer: process.env.WEBSITE_URL
  ? undefined
  : {
    command: "npm run dev --port 8080",
    port: 8080,
    reuseExistingServer: true,
  },

Then, update our playwright.yaml workflow to run tests against the deploy preview.
To wait for the Netlify deploy preview URL, we can use the mmazzarolo/wait-for-netlify-action GitHub Action.

mmazzarolo/wait-for-netlify-action is a fork of probablyup/wait-for-netlify-action. By default, probablyup/wait-for-netlify-action assumes it's running within a workflow triggered by a pull request push. In our case, the update-snapshots.yml workflow is triggered by a comment, so I forked this GitHub Action to ensure it runs on any workflow, regardless of what triggered it.

The wait-for-netlify-action GitHub Action requires two things:

Setting a NETLIFY_TOKEN GitHub Action secret with a Netlify Personal Access Token.
Passing a Netlify Site ID (from Netlify: Settings → Site Details → General) to the site_id parameter in the workflow.

.github/workflows/playwright.yaml

 name: Playwright Tests
 
 on:
   push:
     branches: [main, master]
   pull_request:
     branches: [main, master]
   workflow_run:
     workflows: ["Update Snapshots"]
     types:
       - completed
 
 jobs:
   test:
     timeout-minutes: 60
     runs-on: ubuntu-latest
     steps:
       # Checkout and setup
       - uses: actions/checkout@v2
       - uses: actions/setup-node@v2
         with:
           node-version: "14.x"
       - name: Install dependencies
         run: npm install
       - name: Install Playwright Browsers
         run: npx playwright install --with-deps chromium
+      # Wait for the Netlify preview URL to be ready
+      - name: Wait for Netlify Deploy
+        uses: mmazzarolo/wait-for-netlify-action@8a7a8d8cf5b313c916d805b76cc498380062d268
+        id: get-netlify-preview-url
+        with:
+          site_id: "YOUR_SITE_ID"
+        env:
+          NETLIFY_TOKEN: ${{ secrets.NETLIFY_TOKEN }}
+      # The Netlify preview URL is now available
+      # as `steps.get-netlify-preview-url.outputs.url`
+      - name: Run Playwright tests
-        run: npx playwright test
+        run: WEBSITE_URL=${{ steps.get-netlify-preview-url.outputs.url }} npx playwright test
       # Upload the test result artifacts
       - uses: actions/upload-artifact@v2
         if: failure()
         with:
           name: playwright-report
           path: playwright-report/
           retention-days: 30

.github/workflows/playwright.yaml

 name: Playwright Tests
 
 on:
   push:
     branches: [main, master]
   pull_request:
     branches: [main, master]
   workflow_run:
     workflows: ["Update Snapshots"]
     types:
       - completed
 
 jobs:
   test:
     timeout-minutes: 60
     runs-on: ubuntu-latest
     steps:
       # Checkout and setup
       - uses: actions/checkout@v2
       - uses: actions/setup-node@v2
         with:
           node-version: "14.x"
       - name: Install dependencies
         run: npm install
       - name: Install Playwright Browsers
         run: npx playwright install --with-deps chromium
+      # Wait for the Netlify preview URL to be ready
+      - name: Wait for Netlify Deploy
+        uses: mmazzarolo/wait-for-netlify-action@8a7a8d8cf5b313c916d805b76cc498380062d268
+        id: get-netlify-preview-url
+        with:
+          site_id: "YOUR_SITE_ID"
+        env:
+          NETLIFY_TOKEN: ${{ secrets.NETLIFY_TOKEN }}
+      # The Netlify preview URL is now available
+      # as `steps.get-netlify-preview-url.outputs.url`
+      - name: Run Playwright tests
-        run: npx playwright test
+        run: WEBSITE_URL=${{ steps.get-netlify-preview-url.outputs.url }} npx playwright test
       # Upload the test result artifacts
       - uses: actions/upload-artifact@v2
         if: failure()
         with:
           name: playwright-report
           path: playwright-report/
           retention-days: 30

Finally, we can update the update-snapshots.yml workflow just like we did above.

.github/workflows/update-snapshots.yml

 name: Update Snapshots
 
 on:
   # It looks like you can't target PRs-only comments:
   # https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request_comment-use-issue_comment
   # So we must run this workflow every time a new comment is added to issues and PRs
   issue_comment:
     types: [created]
 
 jobs:
   updatesnapshots:
     # Run this job only on comments of pull requests that strictly match
     # the "/update-snapshots" string
     if: ${{ github.event.issue.pull_request && github.event.comment.body == '/update-snapshots'}}
     timeout-minutes: 60
     runs-on: ubuntu-latest
     steps:
       # Checkout and do a deep fetch to load all commit IDs
       - uses: actions/checkout@v2
         with:
           fetch-depth: 0 # Load all commits
           token: ${{ secrets.GITHUB_TOKEN }}
       # Get the SHA and branch name of the comment's pull request
       # We must use the GitHub API to retrieve these informations because they're
       # not accessibile within workflows triggered by "issue_comment"
       - name: Get SHA and branch name
         id: get-branch-and-sha
         run: |
           sha_and_branch=$(\
             curl \
               -H 'authorization: Bearer ${{ secrets.GITHUB_TOKEN }}' \
               https://api.github.com/repos/${{ github.repository }}/pulls/${{ github.event.issue.number }} \
             | jq -r '.head.sha," ",.head.ref');
           echo "::set-output name=sha::$(echo $sha_and_branch | cut -d " " -f 1)";
           echo "::set-output name=branch::$(echo $sha_and_branch | cut -d " " -f 2)"
       # Checkout the comment's branch
       - name: Fetch Branch
         run: git fetch
       - name: Checkout Branch
         run: git checkout ${{ steps.get-branch-and-sha.outputs.branch }}
       # Setup testing environment
       - uses: actions/setup-node@v2
         with:
           node-version: "14.x"
       - name: Install dependencies
         run: npm install
+      # Wait for the Netlify preview URL to be ready
+      - name: Wait for Netlify Deploy
+        uses: mmazzarolo/wait-for-netlify-action@8a7a8d8cf5b313c916d805b76cc498380062d268
+        id: get-netlify-preview-url
+        with:
+          site_id: "YOUR_SITE_ID"
+          commit_sha: ${{ steps.get-branch-and-sha.outputs.sha }}
+        env:
+          NETLIFY_TOKEN: ${{ secrets.NETLIFY_TOKEN }}
       - name: Install Playwright browsers
         run: npx playwright install --with-deps chromium
+      # Update the snapshots based on the Netlify preview
       - name: Update snapshots
-        run: npx playwright test --update-snapshots --reporter=list
+        run: WEBSITE_URL=${{ steps.get-netlify-preview-url.outputs.url }} npx playwright test --update-snapshots --reporter=list
       # Commit the changes to the pull request branch
       - uses: stefanzweifel/git-auto-commit-action@v4
         with:
           commit_message: "[CI] Update Snapshots"

.github/workflows/update-snapshots.yml

 name: Update Snapshots
 
 on:
   # It looks like you can't target PRs-only comments:
   # https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request_comment-use-issue_comment
   # So we must run this workflow every time a new comment is added to issues and PRs
   issue_comment:
     types: [created]
 
 jobs:
   updatesnapshots:
     # Run this job only on comments of pull requests that strictly match
     # the "/update-snapshots" string
     if: ${{ github.event.issue.pull_request && github.event.comment.body == '/update-snapshots'}}
     timeout-minutes: 60
     runs-on: ubuntu-latest
     steps:
       # Checkout and do a deep fetch to load all commit IDs
       - uses: actions/checkout@v2
         with:
           fetch-depth: 0 # Load all commits
           token: ${{ secrets.GITHUB_TOKEN }}
       # Get the SHA and branch name of the comment's pull request
       # We must use the GitHub API to retrieve these informations because they're
       # not accessibile within workflows triggered by "issue_comment"
       - name: Get SHA and branch name
         id: get-branch-and-sha
         run: |
           sha_and_branch=$(\
             curl \
               -H 'authorization: Bearer ${{ secrets.GITHUB_TOKEN }}' \
               https://api.github.com/repos/${{ github.repository }}/pulls/${{ github.event.issue.number }} \
             | jq -r '.head.sha," ",.head.ref');
           echo "::set-output name=sha::$(echo $sha_and_branch | cut -d " " -f 1)";
           echo "::set-output name=branch::$(echo $sha_and_branch | cut -d " " -f 2)"
       # Checkout the comment's branch
       - name: Fetch Branch
         run: git fetch
       - name: Checkout Branch
         run: git checkout ${{ steps.get-branch-and-sha.outputs.branch }}
       # Setup testing environment
       - uses: actions/setup-node@v2
         with:
           node-version: "14.x"
       - name: Install dependencies
         run: npm install
+      # Wait for the Netlify preview URL to be ready
+      - name: Wait for Netlify Deploy
+        uses: mmazzarolo/wait-for-netlify-action@8a7a8d8cf5b313c916d805b76cc498380062d268
+        id: get-netlify-preview-url
+        with:
+          site_id: "YOUR_SITE_ID"
+          commit_sha: ${{ steps.get-branch-and-sha.outputs.sha }}
+        env:
+          NETLIFY_TOKEN: ${{ secrets.NETLIFY_TOKEN }}
       - name: Install Playwright browsers
         run: npx playwright install --with-deps chromium
+      # Update the snapshots based on the Netlify preview
       - name: Update snapshots
-        run: npx playwright test --update-snapshots --reporter=list
+        run: WEBSITE_URL=${{ steps.get-netlify-preview-url.outputs.url }} npx playwright test --update-snapshots --reporter=list
       # Commit the changes to the pull request branch
       - uses: stefanzweifel/git-auto-commit-action@v4
         with:
           commit_message: "[CI] Update Snapshots"

That should do it. From now on, our Playwright visual regression tests will run against the deploy previews.

Conclusion

I hope this blog post gave you a solid foundation for building your visual testing setup. A few ideas on how you can improve it further:

You'll probably want to test full screen snapshots using the fullScreen parameter instead of just the visible viewport. And maybe capture mobile snapshots too.
If your web application loads parts of the UI asynchronously (e.g., images, videos), you'll probably want to wait for them to be loaded or omit them from the tests.
You can restrict the /update-snapshots command so that it can only be invoked by the repository's owners.
You can trigger the snapshots updating flow from a different source (e.g., webhooks) instead of relying on GitHub comments.
You can store the snapshots on a third-party storage solution.
If you're using deploy previews, you can optimize the workflows by parallelizing the step waiting for the preview link with the rest of the workflow.

Acknowledgments

Across this blog post, I copy-pasted snippets and paragraphs from the Playwright and the Cypress "Funcional VS visual testing" documentation.