How do you run cross browser testing today?
Applitools released a new product and it looks to be rather innovative. Focused on visual testing, Applitools first caught my eye with a 2015 demo which showed how their algorithms could detect the difference between a layout modification and a content modification on a webpage, quite the step up from pixel-perfect image diffing that other early visual UI testing tools were doing.
Their new product, Ultrafast Grid, drastically improves the speed of running a suite of cross browser visual UI tests. Ultrafast Grid improves speed and reduces cost for all teams using visual web testing, but I’m personally excited about the opportunity for promoting visual testing habits directly to frontend developers. Frontend developers can now get instant feedback for their changes across an array of desktop and mobile browsers. Applitools bets that if the total time of visual UI test suites can be reduced to seconds or minutes, they can be fit into the instant feedback cycle employed by modern developers while actively coding. There’s a huge difference between waiting an hour to find out if your commits broke something and waiting a minute. Retaining a developer’s flow-state boosts productivity, and reduces frustration.
Hands-On Experience
I tested a beta version of Applitools Ultrafast Grid and it delivered on its promise. I wrote up a script to visit every single page and article of our Appium Pro newsletter on two different browsers, with three different screen resolutions. The entire process took less than three minutes!
62 pages * 2 browsers * 3 screen sizes
= 372 images captured and verified in less than 3 minutes
Crossing the 1-3 minute mark for an entire cross-browser test suite took considerable technical effort on the part of Applitools, and Ultrafast Grid takes a novel approach that, as far as I’m aware, nobody else is offering. The crux of their innovation is that a single browser is automated to crawl a site locally on the developer’s machine while a fleet of machines in their cloud take screenshots and compare them to past images.
The developer opens up Chrome locally and visits example.com, the Ultrafast Grid client then sends all of the HTML elements, images, text, etc currently displayed in Chrome to the Ultrafast Grid Cloud. The Applitools browsers in the cloud load the same HTML assets and take a screenshot, effectively testing the layout and rendering in every browser and screen size selected. The traditional way of gathering these comparison screenshots is to use a Selenium cloud provider, spinning up an entire automation session on each browser for testing, navigating to example.com, loading the site, and taking a screenshot.
Avoid The Headaches of Cross-Browser Testing
What Ultrafast Grid’s technology allows is to remove many of the processes which take a long time: booting virtual machines and browsers, downloading javascript and CSS assets, and uploading large screenshot files. The developer can quickly run a single browser locally, and get feedback on how those pages would look on a plethora of foreign devices.
The actual user actions being automated on each webpage are only performed locally on a single browser, while screenshots are rendered and taken in the cloud. You shouldn’t have to pay for Applitools’ remote browsers to click on checkboxes and fill in forms, with Ultrafast Grid you only use their cloud resources for rendering the screens generated by the automation script running locally.
When running automated tests on your site, are you concerned with testing whether Firefox correctly scrolls a combo box, or Safari can put placeholder text in a textbox? Of course not, what we’re all testing for is that the site looks right in every browser, and all our controls are visible. Ultrafast Grid cuts out the overhead of testing browser actions, allowing us to run visual regression tests extremely quickly.
For those familiar with the current paradigm on using Applitools along with a cloud of automated browsers, here are three cases which demonstrate the difference when using Ultrafast Grid:
Case Study 1: Random Number Generator
You are testing a single web page which displays a random number on every page load. No two users will receive the same number. Your automation script opens the page locally using Chromedriver and tells Applitools Ultrafast Grid to check the UI on Chrome and Firefox, with 800×600 resolution and also 720×1280 resolution. The Ultrafast Grid takes screenshots on the four browser/resolution combinations. Do all four images contain different random numbers or the same random number?
All four screenshots will show the same random number which was displayed on the developer’s local Chrome instance. The Ultrafast Grid browsers did not visit the web page in question. Instead, a full copy of the HTML elements, styling, images, etc (the HTML Snapshot), is sent to each of the Ultrafast Grid browsers to render.
Case Study 2: IP Address
You are testing a single web page which displays the IP address of the browser. Your automation script opens the page locally using Chromedriver and tells Applitools Ultrafast Grid to check the UI on Chrome and Firefox, with 800×600 resolution and also 720×1280 resolution. The Ultrafast Grid takes screenshots on the four browser/resolution combinations. Do all four images contain different IP addresses or the same IP addresses?
As with Case Study 1, all four screenshots will show the IP address of the developer’s local computer. It’s interesting to note that the site’s server will only log a single HTTP request emanating from the local Chrome instance, the four Ultrafast Grid browsers never send requests to the site.
Case Study 3: Javascript Code Which Detects Browser Type
You are testing a website which includes a known bug on Firefox browsers. To correct this bug for Firefox users, the javascript code checks the browser’s User Agent and executes a different code path in order to fix the issue. Your automation script opens the page locally using Chromedriver and tells Applitools Ultrafast Grid to check the UI on Chrome and Firefox, with 800×600 resolution and also 720×1280 resolution. The Ultrafast Grid takes screenshots on the four browser/resolution combinations. Do the Firefox screenshots contain the known bug or does the alternate code path get executed?
The Firefox screenshots contain the buggy behavior. The HTML snapshots sent to Ultrafast Grid show the same exact elements on the local developer’s screen at the time of the snapshot. Javascript on the page does not get rerun for each of the Ultrafast Grid Browsers.
As Case Study 3 demonstrates, the speed we gain from using Ultrafast Grid comes at a cost. There exists a class of bugs which Ultrafast Grid will not catch: cross browser incompatibility of the javascript run on the page. Applitools does not argue that one should replace full end-to-end browser automation tests, but points out that the majority of cross-browser issues have to do with rendering and layout. In the modern day of large frontend frameworks, such as React and Angular, cross-browser bugs tend to be layout and style issues. The framework developers are the ones who mainly need to worry about browser javascript incompatibilities.
Inserting Ultrafast Grid Into The Testing Pyramid
I see Applitools Ultrafast Grid as occupying a new layer in the so-called “Testing Pyramid”.
As we move up the pyramid, tests take longer to run and are less reliable but exercise the entire software stack more rigorously. Visual UI tests using Applitools Ultrafast Grid are much faster and less flakey than full browser automation tests, and this translates to cost savings for users. I love how Ultrafast Grid raises the bar for tests which can provide feedback to frontend developers while they are actively coding.
The only drawback is that Ultrafast Grid won’t catch rare issues related to cross-browser _javascript_ execution and race conditions, for these you’ll still want some form of full browser automation. But you don’t need to test 100% of your functionality with full end-to-end automation tests. Don’t forget how the pyramid tapers towards the top: when you move many of your current end-to-end tests to use Ultrafast Grid, you make available more time to test specific browser differences, while the Ultrafast Grid tests runs cheaply on a greater diversity of browsers.
For more detail, including a great explanation of the Testing Pyramid and more detail on how Ultrafast Grid is implemented, see this Webinar by Applitools Architect Gil Tayar. I support Gil’s opinion that in order for a software organization to produce a high-quality product and be able to iterate quickly, developers need to write and run their own tests. Applitools Ultrafast Grid enables frontend developers to work more closely with their visual UI testing.
Want To Know More About Applitools?
Find out more about Applitools. Setup a live demo with us, or if you’re the do-it-yourself type, sign up for a free Applitools account and follow one of our tutorials.
Other reading for you:
- Applitools Appium Native Java Tutorial
- How To Do Cross-browser Visual Testing with Selenium
- Applitools FAQ – Cross Browser Testing by Yarden Naveh
- Applitools Cross Browser Testing Blog Posts
