How To Do Cross-browser Visual Testing with Selenium

A Solution

By leveraging a solution like Applitools Eyes we can reliably run the same visual tests against multiple browsers (which we can gain access to through the use of Sauce Labs).

A Visual Matching Primer

Normally, you would use a strict visual match for visual regression testing. This is good for verifying the layout of an application in the same execution environment (e.g., same browser, screen size, device, etc.), and with exactly the same (or very similar) content. And if you want to cover multiple execution environments (e.g., multiple browsers, various screen sizes, etc.) you need to maintain several baseline images.

But with layout matching (a feature that Applitools Eyes offers) we can use a single baseline for validating multiple execution environments and/or sites with extremely dynamic content.

Let’s dig in with an example.

An Example

NOTE: This example builds on the test code from this previous write-up which covers how to add visual testing to existing Selenium tests using Applitools Eyes and Sauce Labs. You can see the full code example from it here. In order to follow along with this example, you’ll want to familiarize yourself with the sample code. Also, if you want to play along at home, you’ll need to have an account for both Applitools Eyes and SauceLabs (they have free trial options).

To prepare ourselves for cross-browser testing, we’ll need to modify our setup() method. This is where we’ll focus all of our efforts for this post.

Here is where we left off with the setup() method from the previous post.

// filename: Login.java
// ...
    @Before
    public void setup() throws Exception {
        DesiredCapabilities capabilities = DesiredCapabilities.internetExplorer();
        capabilities.setCapability("platform", Platform.XP);
        capabilities.setCapability("version", "8");
        capabilities.setCapability("name", testName);
        String sauceUrl = String.format(
                "http://%s:%s@ondemand.saucelabs.com:80/wd/hub",
                "YOUR_SAUCE_USERNAME",
                "YOUR_SAUCE_ACCESS_KEY");
        WebDriver browser = new RemoteWebDriver(new URL(sauceUrl), capabilities);
        sessionId = ((RemoteWebDriver) browser).getSessionId().toString();
        eyes = new Eyes();
        eyes.setApiKey("YOUR_APPLITOOLS_API_KEY");
        driver = eyes.open(browser, "the-internet", testName);
    }
// ...

First, let’s modify the DesiredCapabilities instantiation so that we can more flexibly specify the browser name.

   @Before
    public void setup() throws Exception {
        DesiredCapabilities capabilities = new DesiredCapabilities();
        capabilities.setCapability("browserName", "firefox");
        capabilities.setCapability("platform", Platform.XP);
        capabilities.setCapability("version", "36");
        capabilities.setCapability("name", testName);

Next, we’ll want to specify the name of the baseline so we can reuse it. If one doesn’t exist, it will be created. And if one exists, it will be used for comparison. This enables us to run our test in one browser, capture a baseline, and then rerun the same test against another browser, and compare the results between the browsers. And if we don’t do it, a separate baseline will automatically be created and used for each browser or screen size.

       eyes.setBaselineName(testName);
        driver = eyes.open(browser, "the-internet", testName);
    }

Now let’s save the file and run the test to capture the baseline (e.g., mvn clean test -Dtest=Login.java from the command line).

Once the test completes, Applitools will provide the URL to the job in the test output. You can review it to make sure it is what you expect. If it is Accept and Save it. Alternatively, you can assume the baseline image is correct and proceed without checking it. It will automatically be used as the baseline on future test runs.

Now that we have a baseline, let’s change our browserName and version capabilities so our test will run against a different browser.

   @Before
    public void setup() throws Exception {
        DesiredCapabilities capabilities = new DesiredCapabilities();
        capabilities.setCapability("browserName", "internet explorer");
        capabilities.setCapability("platform", Platform.XP);
        capabilities.setCapability("version", "8");
    // ...

When we save and run it (e.g., mvn clean test -Dtest=Login.java from the command-line) the test fails.

When we view the results we can see that there are no visual bugs between Firefox and Internet Explorer. Instead, there are differences in how the page was rendered (e.g., different screen sizes, different placement of text, etc.).

The test was considered a failure because the Applitools Eyes match level defaults to Strict mode, which performs a match that is very exact. For effective cross-browser visual testing, we’ll need to use a different match level called Layout2.

Let’s update our test code to use this match level instead.

import com.applitools.eyes.MatchLevel;

    // ...

        eyes.setBaselineName(testName);
        eyes.setMatchLevel(MatchLevel.LAYOUT2);
        driver = eyes.open(browser, "the-internet", testName);
    }

Now when we save our test and run it again (e.g., mvn clean test -Dtest=Login.java from the command-line) it will pass.

For Consistent Results

The test worked this time, but for consistent results we’ll want to specify the viewport size in our Applitools setup. This will help ensure that the viewport size of each browser is consistent regardless of the browser used and the system’s screen resolution.

If you don’t know what size to specify, go with a generic value like 1000x600.

import com.applitools.eyes.RectangleSize;

  // ...

        eyes.setBaselineName(testName);
        eyes.setMatchLevel(MatchLevel.LAYOUT2);
        driver = eyes.open(browser, "the-internet", testName, new RectangleSize(1000, 600));
    }

If the application you’re testing is responsive you can verify it’s layout by changing the viewport size to force the page layout to change. But regardless of the viewport size specified, Applitools Eyes will scroll through the page and stitch together a full page screenshot for validation on each test run.

A Small Bit of Cleanup

Rather than constantly modifying our capabilities by hand to change the browser, version, and platform let’s update the test setup to retrieve runtime properties specified on the command-line instead.

   @Before
    public void setup() throws Exception {
        DesiredCapabilities capabilities = new DesiredCapabilities();
        capabilities.setCapability("browserName", System.getProperty("browser", "firefox"));
        capabilities.setCapability("platform", System.getProperty("platform", "Windows XP"));
        capabilities.setCapability("version", System.getProperty("browserVersion", "36"));
    // ...

With this approach we’re also able to set sensible defaults, which we’ve done. So now if we don’t specify anything, Firefox 36 will run on Windows XP.

To specify values we need to use the -D flag when running our tests on the command line. Here are some examples of it in use. Notice the use of double-quotes for values with spaces.

mvn clean test -Dtest=Login.java -Dbrowser="internet explorer" -DbrowserVersion=8
mvn clean test -Dtest=Login.java -Dbrowser="internet explorer" -DbrowserVersion=10 -Dplatform="Windows 8"
mvn clean test -Dtest=Login.java -Dbrowser=firefox -DbrowserVersion=26 -Dplatform="Windows 7"
mvn clean test -Dtest=Login.java -Dbrowser=safari -DbrowserVersion=8 -Dplatform="OS X 10.10"
mvn clean test -Dtest=Login.java -Dbrowser=chrome -DbrowserVersion=40 -Dplatform="OS X 10.8"

For a full list of available browser and operating system combinations, check out Sauce Labs’ platform list.

To read more about Applitools’ visual UI testing and Application Visual Management (AVM) solutions, check out the resources section on the Applitools website. To get started with Applitools, request a demo or sign up for a free Applitools account.

Tagged

Cross-browser TestingDave HaeffnerSauce LabsSeleniumTest EngineersVisual Testing StrategiesVisual Testing Tools

Dave Piacente