Best practices for optimal visual tests results

Eyes finds visual differences by comparing a checkpoint image to a baseline image and reporting differences that are significant, meaning differences that are noticeable to the human eye. In some applications, the nature of the content or the layout may lead to differences that are not significant from the application point of view. Eyes provides various tools that allow you to tweak the image capture and matching to deal with challenging page content or layout, or with special application requirements.

This article helps you optimize your test flow by describing scenarios in which Eyes reports mismatches when using the default configuration, and by describing how you can customize Eyes so that only relevant mismatches are reported.

Overview

Eyes provides various SDK configuration options and Test Manager tools that allow you to change how it captures images and compares checkpoint and baseline images. Customizing Eyes can save you time by allowing you to focus on differences that are significant for your application.

In order to decide what customization is required for a given situation, it is to useful to consider two questions:

• Was the checkpoint image captured correctly?

• Is the comparison of the checkpoint and baseline image appropriate?

This article is organized into two sections.

• Customizing checkpoint capture

• Customizing checkpoint/baseline matching

The first section describes situations where configuring the image capture can improve the results. The second section describes situations in which configuring the way matching is done can improve the results.

Customizing checkpoint capture

The list below describes some common scenarios related to checkpoint capture:

• The checkpoint only includes what is visible in the browser viewport and does not include the content of the page that you can see by manually scrolling on the page.

  

  Eyes allows you to control whether the checkpoint image should include only the viewport (i.e. what you see in the browser window when you first load a page), or whether it should include the full page (i.e. what you would see if you manually scrolled down or across a page that is larger than the viewport). You can set whether you use full-page mode for an entire test suite, for a specific checkpoint, or for a series of checkpoints.

  • Entire test suite: In the initial suite configuration, add the Configuration.setForceFullPageScreenshot method:

  • Specific checkpoint: Execute the checkpoint using the Eyes.check method with the fully method:

  • Series of checkpoints: Configure Eyes before the first checkpoint execution using the Eyes.setForceFullPageScreenshot method to change the mode before the series of checkpoints, and again after the checkpoints to return to the previous value:

  When using the Ultrafast Grid, the default is full-page capture. When not using the Ultrafast Grid, the default is viewport capture.

• The page is being scrolled and the entire page is captured, but some elements are missing, duplicated, or overlapping.

  

  When images are captured on a desktop browser (as opposed to when using the Ultrafast Grid), the full page is obtained by using the browser to capture images of the page content at different offsets and stitching these images together to form a single image of the entire page content. The Eyes Stitch mode determines the way this is done, and some stitching issues can be solved by using a different stitch mode.

  When a checkpoint is captured in full-page mode, you can choose between the following stitching methods:

  • StitchMode.CSS recommended in most cases
  • StitchMode.SCROLL (default) recommend if, for any reason, using StitchMode.CSS results in an inappropriate capture result.

  You can set the value you require for an entire test suite or a series of checkpoints.

  • Entire test suite: In the initial suite configuration, add the Configuration.setStitchMode method:

  • Series of checkpoints: Configure Eyes before the first checkpoint execution using the Eyes.setStitchMode method to change the mode before the series of checkpoints, and again after the checkpoints to return to the previous value:

  The Ultrafast Grid has a different mechanism for obtaining a full page, so the stitch mode setting is ignored when running using the Ultrafast Grid.

• The page contains multiple scrolled regions, each of which needs to be scrolled to capture and validate all of the page content.

  

  If your page includes multiple scrolled regions, you should validate each of these as a separate checkpoint. You do this by using the SDK to execute an independent checkpoint on each region.

  • Specific checkpoint: Execute the checkpoint using the Eyes.check method with the region method:

• Small, random mismatches are reported, typically near input fields.

  

  A blinking cursor can cause mismatches, since sometimes it is present in the captured image and sometimes it isn't.

  You can configure Eyes to disable the cursor before capturing an image for an entire test suite or a series of checkpoints:

  • Entire test suite: In the initial suite configuration, add the Configuration.setHideCaret method:

    This is the recommended approach.

  • Series of checkpoints: Configure Eyes before the first checkpoint execution using the Eyes.setHideCaret method to change the mode before the series of checkpoints, and again after the checkpoints to return to the previous value.

  Hiding the cursor before capture is the recommended way of eliminating cursor artifacts. If you are using the image SDKs (that take a pre-captured image), or if hiding the cursor is not supported by your browser, then an alternative way is to ask Eyes to detect artifacts caused by blinking cursors and not report them as mismatches. You can request this by using the method Configuration.setIgnoreCaret or the method Eyes.setIgnoreCaret.

  • Entire test suite: In the initial suite configuration, add the Configuration.setIgnoreCaret

  • Series of checkpoints: Configure Eyes before the first checkpoint execution the by calling the Eyes.setIgnoreCaret method with a value of true. You can subsequently disable ignoring the cursor by calling this with with a value of false.

• Mismatches are found on or near a scroll bar.

  Including scroll bars in the captured image can cause differences to be found, and it's best practice to always hide them.

  Mismatches can occur in some browsers due to inconsistencies in the way scroll bars are displayed. To prevent this from happening, Applitools recommends that you always hide the scroll bars for all tests. Most users will do this for an entire test suite, but you can also do this a series of checkpoints.

  • Entire test suite: In the initial suite configuration, add the Configuration.setHideScrollbars method:

  • Series of checkpoints: Configure Eyes before the first checkpoint execution using the Eyes.setHideScrollbars method to change the mode before the series of checkpoints, and again after the checkpoints to return to the previous value.

• Elements are missing in the captured image. This may happen because the checkpoint is captured before the page is fully loaded, and is therefore not ready for validation, or if the page implements "lazy loading" and only updates the page when it is scrolled.

  

  If one or more elements are missing in a captured checkpoint image, it may be because the application takes a significant amount of time to update the browser window, and the checkpoint is executed before the window is fully updated. This can happen, for example, because animations have not completed or because displayed data is being obtained from remote servers and has not yet been received when the checkpoint is executed.

  The best way to avoid such issues is to use your browser infrastructure (e.g. Selenium/Appium) to wait until the page is fully displayed. As a backup solution, Eyes provides two techniques:

  • Adding a fixed time delay before a screen capture to give the display time to update.
  • Retrying capture of a checkpoint if there are mismatches, up to a maximum time limit.

  Eyes provides two mechanisms that can help you solve timing problems where more explicit and synchronous methods can't be used. You can set the value you require for an entire test suite, for a specific checkpoint, or for a series of checkpoints.

  • wait time: The amount of time Eyes waits before it starts to capture a screenshot. This can be increased to give the browser extra time to fully update the window.
  • match timeout: The amount of time that Eyes will continue to retry capturing a screenshot if previous capture attempts resulted in a mismatch. The attempt to retry assumes that the mismatch may be caused by capturing the image too soon, and that on a subsequent attempt the window will be fully updated.

  You can set the default wait time and match timeout in various ways:

  • Entire test suite: In the initial suite configuration, add the Configuration.setWaitBeforeScreenshots or Configuration.setMatchTimeout methods as required.

  • Specific checkpoint: Execute the checkpoint using the Eyes.check method appended with the timeout method.

  • Series of checkpoints: Configure Eyes before the first checkpoint execution using the Eyes.setWaitBeforeScreenshots or Eyes.setMatchTimeout methods as required. Then use them again after the series of checkpoints to set values to the previous value or another value.

  Note that these capture timing-related methods are ignored when using the Ultrafast Grid, since the Ultrafast Grid uses other mechanisms to wait for the page to stabilize.

• A page inlcudes JavaScript that changes the page content based on the viewport size, and unexpected mismatched are found when running using the Ultrafast Grid.

  

  Dealing with viewport size dependent JavaScript is described in the article .Handling viewport-dependent JavaScript.

Customizing checkpoint/baseline matching

By default, the Eyes matching algorithms assume that the checkpoint and baseline images are, in principle, identical (i.e. they contain the same set of elements, the content of each element is the same, and the relative positions of the elements don't change). In some applications, the content may be dynamic, meaning that elements can have different text or graphics in different test runs, or that the size or position of elements can be different in different test runs. Eyes provides a variety of tools to enable visual validation of pages with dynamic content.

The list below describes some common scenarios in which you might need to change the way Eyes does matching in order to obtain relevant match results.

• You are testing a page using A/B testing and getting mismatches every time the A/B test choice does not match the baseline.

  

  Eyes supports visual testing of applications that are undergoing A/B testing by allowing checkpoints to be matched against a baseline that includes multiple variations of the baseline image per checkpoint. Eyes considers a checkpoint image to be matched if it matches any variation of the baseline image. This feature is described in detail in the article Working with baseline variations .

• The text or graphics are dynamic and can be different every test run. For example, a field that contains a date, or a targeted ad that is evaluated at run time, can create a mismatch with the value in that field in the baseline image. You want to validate the layout of the fields but not the dynamic content that changes.

  

  Eyes allows you to configure the Match level, which controls the type of image matching done when comparing checkpoint and baseline images. The main match points are:

  • Strict: This match level reports differences that are visible to the human eye. It is the default match level.
  • Layout: This match level identifies the different types of content areas (such as text or images) and their position relative to each other. It then reports differences in the layout, ignoring differences in the actual content of the text and images.

  If your dynamic content has a static layout and only the text or graphics change dynamically, then you can use a Layout match level so that mismatches are reported for changes in layout but not for differences in the text or graphics content. For more details on the available match levels see How to use Eyes match levels.

  You can set the match level using either the SDK or the Test Manager. In the SDK, you can set the match level for an entire test suite, for a series of checkpoints, for a specific checkpoint, or for specific regions or elements in that checkpoint. In the Test Manager, you can set the match level for one or more regions in a step and then copy them to other similar steps.

  • Test Manager for a given step: You can define match level regions by using the Test editor or Step editor to mark rectangular areas on the checkpoint or baseline image. These match levels are applied after you save the baseline and run the test again. See Add special regions to the Step baseline and checkpoint.

    The Test Manager also allows you to change the match level on the results of the current step to see how a change in match level would affect the mismatch reported. This does not change the outcome of the current test or future tests – it is a “what if” analysis tool. See Preview the step images using match level options for details.

  • SDK - Entire test suite: In the initial suite configuration, add the Configuration.setMatchLevel method:

  • SDK - Specific checkpoint: Execute the checkpoint using the Eyes.check method together with the matchLevel method:

  • SDK - Specific region(s) in a checkpoint: Use the Eyes.check method along with one or more calls to the strict, ignorecolors, or exact methods – one call for each match level region:

  • SDK - Checkpoint and specific checkpoint region(s): Combine setting one match level for the entire checkpoint using matchLevel and setting specific match levels for one or more regions:

    Note that by providing a class element selector (input-fields), you can define multiple regions (i.e. all regions that are selected by that class).

  • Series of checkpoints: Configure Eyes before the first checkpoint in the series using Eyes.setMatchLevel to set the required level for the series of checkpoints, and again after the series of checkpoints to set the required level for the checkpoints that follow.

  The advantage of using the Test Manager to define a region is that you can do so interactively while viewing the mismatch. You don't need to access the code. You can also ask Eyes to find similar regions in the test or batch and add the region there as well. For more details see Adding, modifying, or deleting regions of a step.

  The advantage of using the SDK to define a region is that you can specify the region using an element object or an element selector. If the selector can select several elements (e.g. a class selector), then a region will be created for each element selected by the class. Using a selector is also more flexible since the region remains well defined even if the position or size of the element changes over time. If required, you can also define a rectangular area in a given absolute position and of a given size.

• The position of an element is dynamic and can change in different test runs, but the move is reported as a mismatch. The maximum the element can move in any direction is known.

  

  If the position of an element changes relative to the baseline, then by default Eyes flags it as a mismatch. You can define a floating region to indicate that a particular region or element may move by up to a maximum amount and that this should not be reported as a mismatch.

  A floating region defines two regions: an inner region and an outer region that fully encloses the inner region.

  

  The inner region marks a rectangular area of interest that may change position in different test runs. The outer region marks the extent to which it can move in each direction without being considered a mismatch. Formally, all pixels of the checkpoint outer region will be considered as matched if the pixels of the checkpoint inner region match at least one area in the baseline outer region.

  You can define floating regions using either the SDK or the Test Manager. In the SDK you can define floating regions for a specific region or elements in a checkpoint. In the Test Manager you can define floating regions in a specific step and for visually similar regions and steps.

  • Test Manager for a given step:: You can define floating regions using the Test editor or Step editor by marking a rectangular area on either the checkpoint or the baseline image. After you save the baseline, these floating regions are used in subsequent tests on that baseline. For more information see Add special regions to the Step baseline and checkpoint.

  • SDK - Regions in a specific checkpoint: You can define one or more floating regions on a checkpoint using the class$eyes$check method along with one or more calls to the floating method.

  The advantage of using the Test Manager to define a region is that you can do so interactively while viewing the mismatch. You don't need to access the code. You can also ask Eyes to find similar regions in the test or batch and add the region there as well. For more details see Adding, modifying, or deleting regions of a step.

  The advantage of using the SDK to define a region is that you can specify the region using an element object or an element selector. If the selector can select several elements (e.g. a class selector), then a region will be created for each element selected by the class. Using a selector is also more flexible since the region remains well defined even if the position or size of the element changes over time. If required, you can also define a rectangular area in a given absolute position and of a given size.

• A change in size of an element causes many other elements to move, and these moves are reported as differences. You are only interested in getting reports of elements that have appeared, disappeared, or changed but do not want to get mismatch reports regarding elements that have only moved.

  

  Eyes can differentiate between image features that are different and image features that have only moved. In the Test Manager, you can use the Hide displacement tool to toggle between seeing all mismatches and hiding mismatches that are only due to displacement. Using the Test Manager tool does not affect the outcome of the test. You can use the SDK to configure Eyes so that it will detect and ignore displaced image features either for the entire test suite or for specific checkpoints.

  • Test Manager for a given step: The Hide displacement diffs tool in the Test editor can be used to toggle whether displacements are displayed for a particular step or not. This has no effect on the result of the test. For more details see Hide displacement diffs tool.

  • SDK - entire test suite: In the initial suite configuration add the Configuration.setIgnoreDisplacements method.

  • SDK - specific checkpoint: Execute the checkpoint using the Eyes.check method with the ignoreDisplacements method.

  When ignoring displacements is enabled using the SDK, Eyes will only fail the step if there are mismatches that are not displacements. In the Test Manager, the hide displacements icon will indicate that ignore displacements is on. You can toggle it off to see what the results look like if displacements are not ignored, but this will not affect the step or test status.

  Ignore displacements only works when using a match level of Strict or Content.

• The page contains text or graphics that you don't want to validate, for example, because they are dynamic, or because they contain animated content such as video.

  

  Eyes allows you to specify regions or elements that should be ignored (i.e. mismatches should not be reported in those areas).

  You can define ignore regions to instruct Eyes that it should not report mismatches in a specified area in the checkpoint image. This can be useful for areas that contain dynamic data such as animated GIFs or dynamic ads, where both the content and the structure of the area are dynamic, and using a Layout match level would find mismatches that are not significant for your application. You can set ignore regions using either the SDK or the Test Manager. In the SDK you can define ignore regions for a specific region or elements in a checkpoint. In the Test Manager you can define ignore regions in a specific step and for visually similar regions and steps.

  • Test Manager for a given step: You can define ignore regions by using the Test editor or Step editor to mark rectangular areas on the checkpoint or baseline image. These will be applied to future tests after you save the baseline and run the test again. See Add special regions to the Step baseline and checkpoint.

  • SDK - regions in a specific checkpoint: You can define one or more ignore regions in a specific checkpoint by using the eyes$check method along with one or more calls to the ignore method:

  The advantage of using the Test Manager to define a region is that you can do so interactively while viewing the mismatch. You don't need to access the code. You can also ask Eyes to find similar regions in the test or batch and add the region there as well. For more details see Adding, modifying, or deleting regions of a step.

  The advantage of using the SDK to define a region is that you can specify the region using an element object or an element selector. If the selector can select several elements (e.g. a class selector), then a region will be created for each element selected by the class. Using a selector is also more flexible since the region remains well defined even if the position or size of the element changes over time. If required, you can also define a rectangular area in a given absolute position and of a given size.