Wraith - Visual Regression Testing

At Red Hat we have a static copy of our site which is configured as a failover in case our main site goes kaboom.

One issue we’ve had is how to verify the failover content is up to date with the live site.

I recently setup Wraith to automate these checks for us.

Wraith is an open-source project created by the BBC.

Wraith uses a headless browser to create screenshots of webpages on different environments (or at different moments in time) and then creates a diff of the two images; the affected areas are highlighted in blue.

You can configure it in various ways but in a nutshell you provide it with two URLs, and it will take screenshots of both and compare them, providing you with a visual diff of the two as well as a ‘score’ percentage of the differences (if any).

Installation

Setup is easy.

You will need ImageMagick installed, as well as a headless browser like PhantomJS.

Then simply:

:::text
gem install wraith

If you run ‘wraith’ without any options it will print out some useful help:

:::text
Commands:
  wraith capture [config_name]              # Capture paths against two domains, compare them, generate gallery
  wraith compare_images [config_name]       # compares images to generate diffs
  wraith copy_base_images [config_name]     # copies the required base images over for comparison with latest images
  wraith crop_images [config_name]          # crops images to the same height
  wraith generate_gallery [config_name]     # create page for viewing images
  wraith generate_thumbnails [config_name]  # create thumbnails for gallery
  wraith help [COMMAND]                     # Describe available commands or one specific command
  wraith history [config_name]              # Setup a baseline set of shots
  wraith latest [config_name]               # Capture new shots to compare with baseline
  wraith multi_capture [filelist]           # A Batch of Wraith Jobs
  wraith reset_shots [config_name]          # removes all the files in the shots folder
  wraith save_images [config_name]          # captures screenshots
  wraith setup                              # creates config folder and default config
  wraith setup_folders [config_name]        # create folders for images
  wraith validate [config_name]             # checks your configuration and validates that all required properties exist
  wraith version                            # Show the version of Wraith

Before running however you will need to configure two files.

Create a project directory: myproject, navigate into it, then run:

 wraith setup

This will setup a myproject/config folder and a myproject/javascript folder containing some example scripts to help you get started.

:::text
create  configs
create  configs/capture.yaml
create  configs/history.yaml
create  configs/spider.yaml
create  javascript
create  javascript/cookies_and_headers--casper.js
create  javascript/cookies_and_headers--phantom.js
create  javascript/disable_javascript--casper.js
create  javascript/disable_javascript--phantom.js
create  javascript/interact--casper.js
create  javascript/interact--phantom.js
create  javascript/wait--casper.js
create  javascript/wait--phantom.js

I am not going to delve too deeply into configuration. Check out the excellent Wraith documentation.

Of particular interest is the ability to call Javascript before capture. At Red Hat we use this to pass some special headers so we can view failover content.

Example

We’ll run Wraith with the capture command and pass it our config file:

:::text
wraith capture configs/capture.yaml

When you run Wraith it will visit the page, snap a screenshot and then assemble a gallery for you to view.

Here is an example screenshot which shows that in failover we are missing some font files (highlighted in blue).

Jenkins

With Wraith setup the next step is to automate it!

I first setup a job in Jenkins to run Wraith:

:::text
#!/bin/sh
echo "Starting Failover Test"
echo "Checking: http://url1"
echo "Against: http://url2"
echo "Clean up shots directories"
/usr/local/share/gems/gems/wraith-3.1.2/bin/wraith reset_shots /opt/wraith/configs/capture
echo "Running wraith"
/usr/local/share/gems/gems/wraith-3.1.2/bin/wraith capture /opt/wraith/configs/capture.yaml

First we run a Wraith command which cleans up our screenshots directory, then we run Wraith capture.

Next we need to figure out if there are any differences with our images.

To do that we need to tweak our capture.yaml file and adjust the threshold.

:::text
# (optional) The maximum acceptable level of difference (in %) between two 
# images before Wraith reports a failure. Default: 0
threshold: 5

While testing most of my images were only of 3-4 percent different. The example above is 4.14%.

We can lower this threshold value and when running in Jenkins any images below that threshold will flag that build as a failure.

:::text
threshold: 3

On failure I simply send an email with a link to the gallery.

Now we can be alerted of any discrepencies with our failover and have a visual clue on what exactly is different.