07.03.2025

Finding CSS Side Effects in the Gitlab CI/CD Pipeline

Visual Regression with Lost Pixel and Gitlab

When processing CSS rules, side effects are often noticed too late. To find these, Visual Regression Tests are helpful. Lost Pixel is a popular, simple, and very good tool to perform these types of tests. Lost Pixel comes with GitHub Actions support - the integration into Gitlab is not entirely straightforward. We'll show how to use Lost Pixel and Gitlab CI/CD together.

Visual Regression with Lost Pixel and Gitlab

What is Visual Regression Testing?

Visual Regression Testing enables the identification of optical changes based on a previously established baseline. These changes sometimes include regressions, which can be quickly and cost-effectively recognized. Modern applications often contain a lot of JavaScript and CSS. Changes to the source code, or even simple updates, can lead to side effects. To uncover these, Visual Regression Testing is very suitable.

How does Visual Regression Testing work?

The procedure for VRT tools is always similar:

A set of components or pages is defined to be tested.
Based on this, a baseline is generated.
During code changes and updates, screenshots are taken and compared with the baseline.

Often, testing tools work with a threshold. This means the deviation from the baseline has a tolerance - absolute or relative. For example, one could specify that 10 pixels or 1% deviation from the baseline is acceptable.

Why is this the case? - Website rendering differs partially between browsers and operating systems. Here, a certain tolerance is helpfully expressed.

Lost Pixel in Practice

Lost Pixel is a tool from the company nineLemon. It is cloud-based. The simplest integration into a CI pipeline can be realized with Github Actions. The Lost Pixel Cloud Application takes over the management of the baseline. The screenshots of the baseline do not need to be checked into the repository.

(Source: https://lost-pixel.com)

By integrating the provided Github Action, screenshots of the defined pages/components are automatically created and compared with the baseline. The Lost Pixel Cloud Application offers excellent views for the before-and-after comparison.

(Source: https://lost-pixel.com left afterwards, right before)

Over the simple user interface, changes to the baseline can be accepted or rejected. If rejected, the corresponding Pull Request on Github will be marked as "failed".

Lost Pixel with Gitlab

Github Actions is de facto the “first class citizen” implementation of Lost Pixel in CI pipelines. In this section, we would like to show how we use Lost Pixel with the Gitlab software.

Creating the Baseline for Visual Regression Testing

First, the basis is created. For this, the lostpixel.config.js must be set up:

module.exports = {
  pageShots: {
    pages: [
      { path: '/pattern-library/render-pattern/cms/blocks/patterns/text/text.html', name: 'text', threshold: 0.01 },
    ],
    breakpoints: [320, 640, 1024, 1200, 1920],
    baseUrl: 'http://localhost:8000',
  },
  waitBeforeScreenshot: 2000,
  waitForLastRequest: 5000,
  failOnDifference: true,
  shotConcurrency: 1,
  generateOnly: true,
}

This configuration uses "Page Shots". With this, only full-page screenshots are made from the corresponding URLs. A detailed setup of the settings can be found here. The breakpoints parameter allows checking different screen resolutions simultaneously.

To generate and compare the baseline, we use Docker Images to minimize differences through operating systems or browsers. The baseline will be created as follows:

docker run --rm -v $PWD:$PWD -e WORKSPACE=$PWD -e DOCKER=1 -e LOST_PIXEL_DISABLE_TELEMETRY=0 -e LOST_PIXEL_MODE=update --network="host" lostpixel/lost-pixel:v3.22.0

The baseline is thus newly created and stored locally. Since the Lost Pixel Cloud Platform is not usable with Gitlab, these files must be checked into the Git repository.

Installation of Lost Pixel on the Gitlab Runner

First, it must be ensured that the Lost Pixel CLI is available in the pipeline:

apt-get update
apt-get install -y nodejs npm curl
npm i -g lost-pixel
npx playwright@1.47.2 install --with-deps chromium

Playwright is installed in exactly the version needed for Lost Pixel.

Playwright is a modern, Microsoft-developed end-to-end test framework used for testing web applications. It enables automated UI tests in multiple browsers.

Visual Regression Testing in the CI Pipeline

Performing the comparison is now remarkably simple. We start our application (in our case a Django App) and run the comparison command:

python /app/die/manage.py serve --static &
npx lost-pixel local

We ensure that the screenshots are temporarily available to easily examine differences. This allows failed jobs to be easily investigated:

# gitlab-ci.yaml
artifacts:
  paths:
    - .lostpixel/difference/*
    - .lostpixel/current/*
  expire_in: 1 week
  when: always

Here's an excerpt from a possible gitlab-ci.yaml. In this case, we're running Django applications with a temporarily available, local Postgres database:

# gitlab-ci.yaml

stages:
  - lint
  - build
  - test
  - release
  - deploy

# [...]

visual_regression:
  stage: test
  image:
    name: ${TEST_IMAGE_NAME}
    docker:
      user: root
  services:
    - postgres:17-alpine
  before_script:
    - apt-get update
    - apt-get install -y nodejs npm curl
    - python die/manage.py migrate & && npm i -g lost-pixel
    - npx playwright@1.47.2 install --with-deps chromium
  script:
    - python /app/die/manage.py serve --static &
    - npx lost-pixel local
  variables:
  # [...]
  artifacts:
    paths:
      - .lostpixel/difference/*
      - .lostpixel/current/*
    expire_in: 1 week
    when: always

Analysis of the Visual Regression Test

If the job fails, the results can now be easily retrieved from the sidebar in GitLab:

With a click on Search, the folder structure of the artifacts can be searched:

Michael Schilonka LinkedIn

We can also set up visual regression tests for your app.

Get in touch

What alternative tools are there for Visual Regression Testing?

Besides Lost Pixel, the tool which we primarily use at Blueshoe for automated frontend testing, there are also some alternatives:

Cloud-based Tools

Applittools Eyes – AI-supported visual tests with integration in many test frameworks
Percy (by BrowserStack) – Automated UI Screenshots with GitHub Integration
Visual AI by LambdaTest – Cloud-based visual regression tests
Chromatic – Specifically for Storybook-based UI tests

Open-Source & Self-hosted Tools

BackstopJS – Headless Browser-based visual regression tests
Wraith – Developed by BBC Open-Source tool for screenshot comparison
Resemble.js – JavaScript library for pixel-precise image comparisons
Pixelmatch – Lightweight image comparison library for visual regression

Conclusion

In today's agile development environment, Visual Regression Testing is a crucial component of quality assurance. It helps to detect unintended UI changes early and ensures that new features or bug fixes do not destroy existing design elements.

By using modern tools like Lost Pixel, teams can efficiently integrate visual tests into their CI/CD pipelines and maintain the consistency and user-friendliness of their applications. Especially in the era of complex web applications and responsive design, reliable visual testing is a real game-changer.

Ultimately, Visual Regression Testing not only saves time and costs for manual UI reviews but also contributes to ensuring a perfect user experience – and that across all devices and browsers. 🚀

Frequently Asked Questions

1. What is Lost Pixel and why should I use it with GitLab?

Lost Pixel is an open-source tool for visual regression testing. In GitLab pipelines, it helps detect unexpected UI changes early and prevents faulty builds by stopping the pipeline if visual differences are found.

2. How can I prevent irrelevant changes from failing visual tests?

Use --diff-threshold to ignore minor visual differences
Apply excludeSelectors to ignore dynamic elements like timestamps
Ensure consistent screenshot conditions such as viewport size and theme

3. How can I integrate Lost Pixel with GitLab Merge Requests?

Lost Pixel can be configured to generate visual diffs for each merge request and attach them as comments. This requires a GitLab CI/CD setup or an external bot integration.

4. How do I ensure only manually approved changes update the visual baseline?

Create a dedicated baseline branch to store reference screenshots
Use a CI/CD job that only merges updated screenshots into the baseline branch after manual review

5. Why does Lost Pixel randomly fail in GitLab pipelines?

Unstable screenshots are often caused by random UI elements or animations.
Use the --wait parameter to ensure all UI components are fully rendered before taking screenshots.