Writing visual tests with the Eyes SDK

This article describes how to build a visual test without using the Ultrafast Grid. Subsequent articles in this overview introduce the Ultrafast Grid and explain how to adapt the test described here to work with the Ultrafast Grid.

Visual testing flow

In the article Overview of visual UI testing we described what visual testing is. We will now explain how to write an Eyes visual test using the Eyes SDK.

The heart of the visual test process consists of the following key steps, repeated as many times as required:

Bring the UI to a known state.
Execute a checkpoint:
1. Capture an image of the application window in that state.
2. Compare the checkpoint image to a baseline image, i.e. a previously captured image in that state known to be correct.

In an Eyes visual test, your test code uses a browser or native device driver such as Selenium or Appium to drive the application to a state you want to verify. The test then calls an Eyes SDK check method, which captures an image of the application window and sends it to the Eyes Server for processing.

The Eyes Server receives all the checkpoint images associated with the test and compares them to a baseline which consists of the same sequence of images. The Server sends back a test result which consists of a sequence of steps where each step indicates one of the following possibilities:

Match: The checkpoint image is visually identical to the corresponding baseline image.
Mismatch: The checkpoint image is visually different from the corresponding baseline image. Eyes has found a visual difference.
New: The checkpoint is new; there is no corresponding baseline image
Missing: A checkpoint is missing; there is a baseline image with no corresponding checkpoint.

You can read more about how Eyes compares a series of checkpoint images to baseline images in the article How Eyes compares checkpoints and baseline images.

Once the test has run, you can review the results in the Eyes Test Manager to determine the cause of the differences (Mismatch, New or Missing) that Eyes has detected. As part of the review you can mark areas that indicate a bug that needs to be fixed, or update the baseline to reflect a change in the UI. You can also make adjustments to the way the the way the checkpoint image is compared with the baseline image, for example by ignoring regions with dynamic data. You can read about this process in the article Reviewing test results and updating the baseline.

Coding an Eyes visual test

The following describes a typical test line-by-line, explaining the key concepts supported by the SDK.

The anatomy of a single test

The general pattern of a single test is as follows:

Create an instance of the eyes class.
Call the eyes$open method so that the SDK can initialize the connections with the browser driver and the Eyes Server.
For each required state, use the browser driver to set up that state and then call one of the check() methods to execute a checkpoint.
After all the checkpoints are complete, call the eyes$closeasync method to indicate to the Eyes Server that the test has completed.

In summary, a "test" in Eyes terminology refers to the sequence of check() calls made between the call to eyes$open and the call to eyes$closeAsync.

Running multiple tests with the Runner class

Often, a test program consists of multiple tests, we will refer to this as a test suite.

Recommended best practice is to use a new Eyes instance for each test. However, if necessary, you can run several tests one after the other using the same Eyes instance.

Eyes provides an entity called a Runner that manages access to Eyes services by multiple Eyes instances. You create a Runner before you create any of the Eyes instances, and after you run all the tests, you call the runner again to retrieve all the test results.

In the sections that follow, we focus on a single test, but we structure the code in a way that allows it to evolve naturally to support multiple tests in a test suite.

Setting up a test run

First, to create a runner object that will be associated with all the Eyes instances, create a classicrunner, which captures raw screenshots from the application under test:

The visualgridrunner is another type of runner that interacts with the Eyes Ultrafast Grid server to render the checkpoint images in the cloud, greatly reducing test execution time as well as allowing rendering on a variaty of simulated and emulated environments. You can read more about the Ultrafast Grid in the article Introduction to the Ultrafast Grid. The code illustrated here is constructed in a way that makes it easy for you to transition to using the Ultrafast Grid in the future.

Next, set up configurations that are common to all tests using a configuration object:

Note the use of the fluent API style. Any configuration values that are not explicitly specified will have a default value.

Following is a brief explanation of each of these configurations. For more details, see Test suite configuration and the relevant API documentation.

configuration$setapikey To run an Eyes tests, you need an API Key. For details about obtaining an API key, see How to obtain your API key. If you assign the key to an environment variable called APPLITOOLS_API_KEY then the SDK automatically use its value, and you don't need to configure it in the test.
configuration$setserverurl The SDK needs the URL of the Eyes Server. If you have a dedicated cloud, or on-premise server, then use this method to define the URL of your server to the SDK. If you use the public cloud, then you don't need to configure the server URL.
configuration$setappname Every test has a unique identifier that has two parts - the application name and the test name. Typically, you use a single application name for all the tests that are used for testing a specific application. Note that test names must be unique per application name. In this example, we set the application name in the global configuration item using this method, and we configure the test name in each test individually.
configuration$setviewportsize In a responsive web design, different viewport The viewport is the rectangular area in a browser in which you view the page content. If the page is large than the viewport then the application will often have scrollbars to allow you to see the content of the entire page. The viewport is one of the key attributes that impacts the baseline that will be used in a visual test. sizes can result in different page content or layout. If the viewport size is the same for all your tests, then you can specify it here. Otherwise, you should set the viewport size for every test. For more information, see the article Using viewports in Eyes.
configuration$setbatch The batch is a way to collect multiple tests into a singe logical unit whose results appear together in the Eyes Test Manager. For additional information, see "Defining a batch of tests" below.

The application name and viewport, along with the test name that we set when setting up the test, determine the baseline Defines the sequence of images to which the sequence of images captured at checkpoints will be compared. A test can have multiple baselines, each of which is characterized by the execution environment it ran on (operating system, browser and viewport size). A baseline can have instances in multiple branches. that Eyes chooses to compare your checkpoints. Two additional attributes that impact which baseline Eyes chooses are the operating system and browser type that you use to run the test. These are provided automatically by the SDK.

The example also shows some other configuration options that make good defaults, and can be changed on a per-checkpoint basis. Following is a brief description of these options:

configuration$setforcefullpagescreenshot If true, the entire page content is matched and not just the viewport.
configuration$setstitchmode When the entire page is matched, the page is scrolled to access the different parts of the page when generating the checkpoint image. This method defines the mechanism used to do this, and the value passed as a parameter is the recommended setting.
configuration$sethidescrollbars If true, the main scroll bar will be hidden before image capture. This is useful to hide artifacts that sometime cause mismatches with scroll bars.
configuration$sethidecaret If true, the cursor is turned off before image capture. This helps to eliminate artifacts that can cause false mismatches because of a blinking cursor.

In summary, we have created a classicrunner and a configuration object. In the next section, we use these objects to set up the Eyes instance created for each test.

Setting up the Eyes instance for each test

The snippet below shows how to set up the Eyes instance and web driver objects that are required for each test.

Create the eyes instance by calling the constructor, passing it the runner object that was created previously.
Use the eyes$setconfiguration method, passing it the suiteconfig object to apply the Suite Configuration values we set up previously to the Eyes instance. Note that this method sets all the configuration values, either to the values defined explicitly or to the SDK default values. In the next section, we describe how you can change only specific configuration values.

Test specific configuration

A test may need some test-specific configuration. For example, we configured the application name in the global configuration since it is common to all the tests. Now in the test, we need to configure the test name:

Note the sequence to change only one specific configuration value:

Call eyes$getconfiguration to get the current configuration values.
Make changes to the current configuration.
Set the updated configuration using the method eyes$setconfiguration.

Starting the test

Start the test by calling the method eyes$open.

The method takes as a parameter a webdriver object. The method returns a webdriver object that is a wrapper of the driver passed as a parameter. Use the returned webdriver object to make the calls you need to the web driver to simulate the application events. This allows Eyes to know what user actions were performed before each visual checkpoint. This information is used in the test Editor when you play back a test.

Adding checkpoints to your test

The following snippet illustrates a simple test with two checkpoints:

In the simplest case, as is done here, you can check the entire browser window with a call to eyes$checkwindow. If you have tests that need to test only parts of the window, or need other special treatment, then consider using the eyes$check method which supports a fluent API that provides many powerful options. For more information see The Eyes SDK check Fluent API.

The string passed to eyes$checkwindow is a descriptive tag. Eyes does not use this string to identify the checkpoint. You can change the tag, it does not need to be unique, and is optional. However, providing a unique tag for every checkpoint is good practice, since it makes it easier for you to associate a result you see in the Eyes Test Manager to the corresponding checkpoint in the code.

Closing the test

After you have completed all the checks in a test you need to inform the Eyes server that the test has ended:

If the test completed normally, then you call the eyes$closeAsync method. If the test aborted, then you call the method eyes$abortasync, and the status of the test displayed in the Eyes Test Manager is Aborted. How you detect that a test aborted depends on your test infrastructure - see the section Full example at the end of the article for some examples.

Wrapping up the test suite run

After running and closing all the tests, you obtain the results of the tests using the runner method classicrunner$getalltestresults as shown in the exanple below:

The call to classicrunner$getalltestresults returns an iterator that you can use to loop through the test results of each test.
The object returned by the iterator, testResultContainer, supports the following methods or properties:
- testResultContainer$getexception$morp Returns a non-null status value if a failure prevented the test from completion.
- testResultContainer$gettestresults$morp Returns a testresults object which contains the detailed results of each test or a null if an exception occurred and test results are not available.
The classicrunner$getalltestresults method takes an optional Boolean parameter that determines what happens if Eyes finds mismatches in any of the tests. If the parameter is true, or no parameter is passed, then an exception is raised if any test had an exception or mismatch of any sort. If you want your test framework to capture and handle the exceptions in the same way as any other assertion failure in your test, then pass true. If you want to generate a custom report or otherwise process the test result programmatically upon test completion, then pass false as the parameter value and use the returned object to extract the information you need.

Defining a batch of tests

When you view test results in the Eyes Test Manager, the test results are structured by batches and tests, where each batch contains one or more tests. Where tests are logically related, organizing tests into batches makes it convenient to compare results, find common issues, and in general manage the results and status of tests.

By default, when using a runner, all the tests (i.e., every call to eyes$open) executed using the runner are displayed in a default batch. Typically it is preferable to explicitly group batches so that you can give the batch a descriptive name. You do this by creating and using a batchinfo object. In this example, when setting up the test suite, a batchInfo object was created and the batchinfo object was configured using the method configuration$setbatch. Then, when we configured the test, we assigned the suiteconfig object to the Eyes instance. Extracting the relevant lines of code, the following sequence of calls shows how the Eyes object becomes associated with the batch:

For more information about how you can take advantage of batches see Working with test batches. In particular, if your test suite uses multiple runners, or runs tests on multiple machines or processes and you want the tests to be in the same batch, then you need to explicitly set the batch id as described in this article.

Summary

To summarize the process, an Eyes visual test is based on the following main objects:

Eyes: Use an object of this class to execute a single visual test.

You create an Eyesobject for each test, configure it, call the open method to start the test, execute one or more checkpoints by using methods such as check, and finally call the eyes$closeAsync method to tell Eyes that there are no more checks in the test. At this point the Eyes server may still be processing the checkpoints of this test, but your test program can proceed with the next test in parallel (on supported SDKs).
Runner: Use an object of this type when executing multiple Eyes instances in a single run to orchestrate the execution of the entire test suite.

The runner manages the interaction between the Eyes instances and the Eyes servers. You create a runner before all the tests start, run all the tests, and after all the tests been closed, you call the runner to wait for all the tests to complete and to get their results.
Configuration: This is an object that is used to configure Eyes capture and processing activities.

Although you can configure the Eyes object directly, it is convenient to set up a configuration object with the common settings, and then apply it to every Eyes instance before the test starts.

The figure below illustrates the flow of commands when running a test suite, it illustrates that a runner session encloses one or more tests, and each tests includes one or more checkpoints.

Full example

The program shown below puts together everything we've discussed in this article. In most SDKs, the code is structured into setup and wrap-up per test run and per test suite as is usually done in tests that are based on test infrastructures. In the case of Python the example uses Pytest fixtures. For an example that uses the Ultrafast Grid see Writing visual tests that use the Ultrafast Grid

The examples are based on TestNG (Java), Node (C#), Mocha (JavaScript), Pytest (Python), and RSpec (Ruby).

To adapt them to a different test infrastructure, implement the four before/after methods, following the pattern illustrated in this example. In the case of the aftertestinstance method, adapt the assignment to testpassed to check if the test passed or not, using the facilities provided by your infrastructure.

For further information, see our video, Getting Started With Applitools in Three Easy Commands: