The Robot Eyes library

This article is a complete guide to using Eyes with the Robot framework.

The article consists of the following sections:

For more detailed documentation of the Eyes library of implementation the Robot framework see the tutorial Testing web apps using Robot Framework.

An overview of writing a Robot Eyes test

To use Eyes, the Robot test Settings section should include the following:

Copy 
*** Settings ***
Library        SeleniumLibrary
Library        EyesLibrary     runner=selenium_ufg  config=resources/applitools_config.yaml
Test Setup     Setup
Test Teardown  Teardown

The runner option can be set to one of the following values:

  • web

  • web_ufg

  • native_mobile_grid

The value assigned to the config option is the name of a configuration file that provides configuration values that apply to all the checkpoints of the test. For details of the structure and content of this file, see The EyesLibrary configuration file.

Every Eyes test starts with a setup and ends with a teardown. In between the setup and teardown, there can be one or more checkpoints that test a window or part of a window.

Startup

The startup has the following form:

Eyes Open options

Some possible options are described in Startup options below. For details of other options that can be passed to the Open command see The EyesLibrary configuration file.

Teardown

The teardown has the following form:

Eyes Close

The checkpoints take a number of forms depending on what part of the window you want to test (the entire window, a region of the window, or a frame). You can also specify various special regions and other options. There are four basic types of checkpoints:

  • Eyes Check Window – Check an entire window. See Checking a window.

  • Eyes Check Region By Selector – Check a region within a window. See Checking a region .

  • Eyes Check Frame By Selector – Check an HTML frame within a window. See Checking a frame.

  • Eyes Check Target – Check a Target. This can then be followed by the Window, Region or Frame keywords. This form allow the Target to be specified as a preassigned variable.

Viewing the results of the test

When you run an Eyes test, there are two ways to see the results: in the Robot console or in the log. The results will be presented as follows:

Copy 
$ robot  -t CheckWindow  tests/atests/acceptance/web2.robot
==============================================================================
Web2                                                                          
==============================================================================
Check Window                                                          | PASS |
------------------------------------------------------------------------------
result summary {
    all results=TestResultContainer{
 test_results = New test [TestResults(steps=1, matches=0, mismatches=0, missing=0, url='https://eyes.applitools.com/app/batches/0123456789?accountId=absdefg12345~~', accessibility_status=None)]
 exception = None}
    passed=1
    unresolved=0
    failed=0
    exceptions=0
    mismatches=0
    missing=0
    matches=0
}
Web2                                                                  | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Output:  /Users/myUser/Projects/WORK/APPLITOOLS/eyes.sdk.python/eyes_robot/output.xml
Log:     /Users/myUser/Projects/WORK/APPLITOOLS/eyes.sdk.python/eyes_robot/log.html
Report:  /Users/myUser/Projects/WORK/APPLITOOLS/eyes.sdk.python/eyes_robot/report.html

If all of the tests have passed, then a PASS indicator is visible. Otherwise, click on the string after "url=" (in line 9 above) to open the Test Manager.

Startup options

The startup has the following form:

Eyes Open options

A sample of possible value options is as follows:

Copy 
app_name=My application name 
viewport=[width height]
branch_name=Custombranch
hide_scrollbars=true

These options can also be defined in the configuration file. See The EyesLibrary configuration file for a full description of available options.

Checkpoint definition

You can define checkpoints in the following ways depending on the target area in the window you want to check and whether you need any special options:

Terminology

This section describes some parameters that are common for several keywords:

Parameter Represents
[L T W H] A string that contains four numbers in the form [left top width height], that define the top left corner, width, and height of a rectangular region in the window.
selector A string that is in a legal Robot Selenium or Robot Appium library format, for example: 
css:div#foo h1
xpath: //div[@id="foo"]//h1
${element} An element expressed as a variable in the form ${element} where the element is usually assigned using Get WebElement, for example: 
${element}=    Get WebElement    css:div#foo h1 
${element}=    Get WebElement    xpath: //div[@id="foo"]//h1
where one of [L T W H],selector, or ${element}
boolean A string that contains a boolean value of either true or false.
string A string value. The string may consist of multiple words if they are separated by a single space.
tgt A keyword that defines the type of target Window, Region, or Frame.

Checking a window

You can test an entire window using either of the following commands:

Copy 
Eyes Check Window    tag
Eyes Check           Target Window    check settings....

The tag is the name of the step. It can consist of multiple words separated by a single space. The Target keyword allows you to add advanced options that are described in Special regions.

By default, this checks the currently visible portion of the window (the viewport). If you set the configuration option force_full_page_screenshot to true in the configuration file or in the Open command, or use the Fully check setting with the Target keyword, then all the content on the page is captured using multiple captures, scrolling, and stitching of the multiple images.

Checking a region

Use this command to test a rectangular region in a window, for example, a single <div>. The element may be defined using a CSS selector, or by using the Target form and providing a Coordinate, Selector, or Element.

Copy 
Eyes Check    Region        By Selector     selector tag
Eyes Check    Target Region By Coordinate   check settings....
Eyes Check    Target Region By Selector     check settings....
Eyes Check    Target Region By Element      check settings....

The Target keyword allows you to add advanced options that are described in Special regions.

Checking a frame

Use this command to test an iframe embedded in a window. Specify the frame to be checked by providing its index (a number), its name (a string), or a CSS selector.

Copy 
Eyes Check    Frame          By Selector    selector
Eyes Check    Target Frame   By Index       numbercheck settings....
Eyes Check    Target Frame   By Name        stringcheck settings....
Eyes Check    Target Frame   By Selector    selectorcheck settings.... 
Eyes Check    Target Frame   By Element     ${element}      check settings....

The Target keyword allows you to add advanced options that are described in Special regions.

Basic keywords

The check settings can include any of the following keywords once only:

With Name 
Eyes Check    Target   tgt   By   where   With Name   string

Use this to define the name of the checkpoint instead of the tag that is normally found in the simpler version of these commands.

Fully 
Eyes Check    Target   tgt   By   where   Fully   boolean

If this is set to a value of true, then the entire targeted window, region, or frame is checked, not just the viewport. This overrides the global configuration setting force_full_page_screenshot for this checkpoint.

Ignore Displacements 
Eyes Check    Target   tgt   By   where   Ignore Displacements   boolean

Use this to set whether the Test Manager should initially display mismatches for image features that have only been displaced, as opposed to real mismatches.

The Eyes server can distinguish between mismatches that are caused by content that has moved on the page as opposed to content that is new, is missing, or has changed. In the Test Manager, you can choose to hide differences that arise due to displaced content. Use this keyword to set whether the Test Manager should display differences that Eyes identified as displaced features when first displaying a step, or whether the Test Manager should hide these differences.

Note that this setting only governs what is displayed. Eyes still considers a feature that has moved to be a mismatch that needs to be resolved. See Hide displacement diffs tool for more details.

Scroll Root Element 
Eyes Check    Target   tgt   By   where   Scroll Root Element locator

Normally, Eyes selects the most appropriate element to scroll to execute the Fully keyword. You can use the scroll_root_element keyword to specify the element to scroll explicitly.

Timeout 
Eyes Check    Target   tgt   By   where   Timeout int

Use this to define the timeout to use when acquiring and comparing screenshots for this target.

Send Dom 
Eyes Check    Target   tgt   By   Send Dom   bool

Use this to enable or disable sending DOM for this checkpoint. This keyword overrides the global setting of sending/not sending the DOM for this checkpoint. The global setting is set using send_dom option.

Before Render Screenshot Hook 
Eyes Check   tgt   By   where   Before Render Screenshot Hook   string

Use this to supply a JavaScript snippet that should be executed on the Ultrafast Grid before the DOM is rendered.

Special regions

When a checkpoint uses the Eyes Check followed by the Target keyword, you can optionally add any of the options listed below (marked as check settings.... above).

Accessibility Region 
Eyes Check    Target   By   where   Accessibility Region By Coordinates   [L T W H] type
Eyes Check    Target   By   where   Accessibility Region By Element       ${element} type
Eyes Check    Target   By   where   Accessibility Region By Selector      selector type

Use this to define an accessibility region and its type. The type can be IgnoreContrast, RegularText, LargeText, BoldText, or GraphicalObject.

match levels regions 
Eyes Check    Target ... By   where   Before Render Screenshot Hook  string

Use this to supply a JavaScript snippet that should be executed on the Ultrafast Grid before the DOM is rendered.

Eyes Check    Target tgt   By   wherematchlevel   Region By Coordinate s[L T W H] 
Eyes Check    Target tgt   By   wherematchlevel   Region By Coordinate
Eyes Check    Target tgt   By   wherematchlevel   Region By ${element}

Use this to define the type of matching done on the targeted region, where matchtype can be IgnoreColors, Layout, or Strict. See How to use Eyes match levels for a detailed description of the different match levels.

Floating Region 
Eyes Check   Target   tgt   By   where   Floating Region By Coordinates   [L T W H]   up   down  left  right
Eyes Check   Target   tgt   By   where   Floating Region By Element       ${element}   up   down  left  right
Eyes Check   Target   tgt   By   where   Floating Region By Selector      selector     up   down  left  right
Eyes Check   Target   tgt   By   where   Floating Region With Max Offset By Coordinates   [L T W H]   offset
Eyes Check   Target   tgt   By   where   Floating Region With Max Offset By Element       ${element}    offset
Eyes Check   Target   tgt   By   where   Floating Region With Max Offset By Selector      selector      offset

Add one more floating region to this target.

When you define a floating region, you define an inner region and an enclosing outer region. All the pixels of the outer region are considered as matched if the pixels of the checkpoint inner region match at least one area in the baseline outer region. The inner region can be defined using the Coordinate, Element, or Selector keyword and the outer region can be defined either by individual offsets outwards from the inner region in all directions up down left right or by a single value representing the offset in all four directions.

Ignore Region 
Eyes Check    Target tgt   By   where   Ignore Region By Coordinates   [L T W H]
Eyes Check    Target tgt   By   where   Ignore Region By Element        ${element}
Eyes Check    Target tgt   By   where   Ignore Region By Selector       selector

Add one or more ignore regions to this target (i.e. areas whose content matches anything).