Cypress - Visual regression testing

In

Visual regression testing

Plan

  1. Take a screenshot of a component, on baseline version. Store it locally.
  2. Take a new screenshot of the same component, on updated version. Store it locally.
  3. Merge both immages into a single one.

Setup

  1. Install the packages

    1
    2
    npm i cypress
    npm i cypress-plugin-snapshots

Generate images

  1. Create configuration

    • Add it to cypress.json file.

      1
      2
      3
      4
      {
        "baseUrl": "http://localhost:8000/",
        "video": false
      }

  2. Create the test file.

    • Commands and APIs.

      • cy.screenshot(): it can, out of the box, create individual images of components.
      • After Screenshot API: it allows us to rename files, change directories, and distinguish visual regression runs from standard ones.

    • Extend the plugins/index.js file in our plug-ins directory to support the two new run types (baseline and comparison) and keep history. Then, we set the path for our images according to the run type:

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33
      34
      35
      36
      37
      38
      39
      40
      41
      42
      43
      44
      45
      46
      47
      48
      49
      50
      51
      52
      53
      54
      const fs = require('fs')
      const path = require('path')
      module.exports = (on, config) => {
        // Add to config, so they can be accessed in your tests
        config.env.baseline = process.env.BASELINE || false
        config.env.comparison = process.env.COMPARISON || false

        on('after:screenshot', details => {
          // only modify the behavior of baseline and comparison runs
          if (config.env.baseline || config.env.comparison) {
            // keep track of file name and number to make sure they are saved
            // in the proper order and in their relevant folders.
            let lastScreenshotFile = ''
            let lastScreenshotNumber = 0

            // append the proper suffix number, create the folder, and move
            const createDirAndRename = filePath => {
              if (lastScreenshotFile === filePath) {
                lastScreenshotNumber++
              } else {
                lastScreenshotNumber = 0
              }
              lastScreenshotFile = filePath
              const newPath = filePath.replace(
                '.png',
                ` #${lastScreenshotNumber}.png`
              )

              return new Promise((resolve, reject) => {
                fs.mkdir(path.dirname(newPath), { recursive: true }, mkdirErr => {
                  if (mkdirErr) {
                    return reject(mkdirErr)
                  }
                  fs.rename(details.path, newPath, renameErr => {
                    if (renameErr) {
                      return reject(renameErr)
                    }
                    resolve({ path: newPath })
                  })
                })
              })
            }

            const screenshotPath = 
              `visualComparison/${config.env.baseline ? 'baseline' : 'comparison'}`

            return createDirAndRename(details.path
              .replace('cypress/integration', screenshotPath)
              .replace('All Specs', screenshotPath)
            )
          }
        })
        return config
      }

    • We may create a separate custom command and then have that new command take a screenshot after finding the element.

      1
      2
      3
      4
      5
      6
      7
      8
      9
      Cypress.Commands.add("getAndScreenshot", (selector, options) => {
        // Note: You might need to tweak the command when getting multiple elements.
        return cy.get(selector).screenshot()
      });

      it("get overwrite", () => {
        cy.visit("https://example.cypress.io/commands/actions");
        cy.getAndScreenshot(".action-email")
      })

  3. Trigger tests.

    • Update package.json.

      1
      2
      3
      4
      "scripts": {
        "cypress:baseline": "BASELINE=true yarn cypress:open",
        "cypress:comparison": "COMPARISON=true yarn cypress:open"
      }

Merge the screenshots

Setup

imagediff

  • example

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    const imagediff = require('imageDiff')

    var imageA = new Image();
    ImageA.src = "image1-file.png";

    var imageB = new Image();
    ImageB.src = "image2-file.png";

    // Create the image that shows the difference between the images
    var diff = imagediff.diff(imageA, imageB);

    // Create a canvas with the imagediff method (with the size of the generated image)
    var canvas = imagediff.createCanvas(diff.width, diff.height);

    // Retrieve the 2d context
    var context = canvas.getContext('2d');

    // Draw the generated image with differences on the canvas
    context.putImageData(diff, 0, 0);

    // Now you can do whatever you want with the canvas
    // for example render it inside a div element:
    document.getElementById("some-div-id").appendChild(canvas);

Best practices

Recognize the need for visual testing

  • Assertions that verify style properties.

  • If E2E tests become full of assertions checking visibility, color and other style properties, it might be time to start using visual diffing to verify the page appearance.

  • ❌ Incorrect usage:

    1
    2
    3
    4
    // avoid this
    cy.get('.completed').should('have.css', 'text-decoration', 'line-through')
      .and('have.css', 'color', 'rgb(217,217,217)')
    cy.get('.user-info').should('have.css', 'display', 'none')

DOM state

  • Best Practice: take a snapshot after you confirm the page is done changing.

    • ❌ Incorrect usage:

      1
      2
      3
      4
      // the web application takes time to add the new item,
      // sometimes it takes the snapshot BEFORE the new item appears
      cy.get('.new-todo').type('write tests{enter}')
      cy.mySnapshotCommand()

    • ✅ Correct usage:

      1
      2
      3
      4
      5
      6
      7
      // use a functional assertion to ensure
      // the web application has re-rendered the page
      cy.get('.new-todo').type('write tests{enter}')
      cy.contains('.todo-list li', 'write tests')
      // great, the new item is displayed,
      // now we can take the snapshot
      cy.mySnapshotCommand()

Timestamps

  • Best Practice: control the timestamp inside the application under test.

    • ✅ Correct usage:

      1
      2
      3
      4
      5
      const now = new Date(2018, 1, 1)

      cy.clock(now)
      // ... test
      cy.mySnapshotCommand()

Application state

  • Best Practice: use cy.fixture() and network mocking to set the application state.

    1
    2
    3
    4
    5
    // stub network calls with intercept, ensure result
    cy.intercept('/api/items', { fixture: 'items' }).as('getItems')
    // ... action
    cy.wait('@getItems')
    cy.mySnapshotCommand()

Visual diff elements

  • Best Practice: use visual diffing to check individual DOM elements rather than the entire page.
    • ✅ Correct usage: targeting specific DOM element will help avoid visual changes from component “X” breaking tests in other unrelated components.

Component testing

  • Best Practice: use Component Testing Plugins to test the individual components functionality in addition to end-to-end and visual tests.