Wraith - Visual Regression TestingSunday, May 01 2016
At Red Hat we have a static copy of our site which is configured as a failover in case our main site goes kaboom.
One issue we've had is how to verify the failover content is up to date with the live site.
I recently setup Wraith to automate these checks for us.
Wraith is an open-source project created by the BBC.
Wraith uses a headless browser to create screenshots of webpages on different environments (or at different moments in time) and then creates a diff of the two images; the affected areas are highlighted in blue.
You can configure it in various ways but in a nutshell you provide it with two URLs, and it will take screenshots of both and compare them, providing you with a visual diff of the two as well as a 'score' percentage of the differences (if any).
Setup is easy.
You will need ImageMagick installed, as well as a headless browser like PhantomJS.
gem install wraith
If you run 'wraith' without any options it will print out some useful help:
Commands: wraith capture [config_name] # Capture paths against two domains, compare them, generate gallery wraith compare_images [config_name] # compares images to generate diffs wraith copy_base_images [config_name] # copies the required base images over for comparison with latest images wraith crop_images [config_name] # crops images to the same height wraith generate_gallery [config_name] # create page for viewing images wraith generate_thumbnails [config_name] # create thumbnails for gallery wraith help [COMMAND] # Describe available commands or one specific command wraith history [config_name] # Setup a baseline set of shots wraith latest [config_name] # Capture new shots to compare with baseline wraith multi_capture [filelist] # A Batch of Wraith Jobs wraith reset_shots [config_name] # removes all the files in the shots folder wraith save_images [config_name] # captures screenshots wraith setup # creates config folder and default config wraith setup_folders [config_name] # create folders for images wraith validate [config_name] # checks your configuration and validates that all required properties exist wraith version # Show the version of Wraith
Before running however you will need to configure two files.
Create a project directory: myproject, navigate into it, then run:
I am not going to delve too deeply into configuration. Check out the excellent Wraith documentation.
We'll run Wraith with the capture command and pass it our config file:
wraith capture configs/capture.yaml
When you run Wraith it will visit the page, snap a screenshot and then assemble a gallery for you to view.
Here is an example screenshot which shows that in failover we are missing some font files (highlighted in blue).
With Wraith setup the next step is to automate it!
I first setup a job in Jenkins to run Wraith:
#!/bin/sh echo "Starting Failover Test" echo "Checking: http://url1" echo "Against: http://url2" echo "Clean up shots directories" /usr/local/share/gems/gems/wraith-3.1.2/bin/wraith reset_shots /opt/wraith/configs/capture echo "Running wraith" /usr/local/share/gems/gems/wraith-3.1.2/bin/wraith capture /opt/wraith/configs/capture.yaml
First we run a Wraith command which cleans up our screenshots directory, then we run Wraith capture.
Next we need to figure out if there are any differences with our images.
To do that we need to tweak our capture.yaml file and adjust the threshold.
# (optional) The maximum acceptable level of difference (in %) between two # images before Wraith reports a failure. Default: 0 threshold: 5
While testing most of my images were only of 3-4 percent different. The example above is 4.14%.
We can lower this threshold value and when running in Jenkins any images below that threshold will flag that build as a failure.
On failure I simply send an email with a link to the gallery.
Now we can be alerted of any discrepencies with our failover and have a visual clue on what exactly is different.