Working with baselines

Eyes visual inspection is based on comparing images captured during a visual test to images previously stored in a baseline. Since a test can have a different look and feel when it runs on different execution environments (e.g. different devices, browsers, browser sizes and operating systems), Eyes supports having multiple baselines per test, where each baseline represents the look and feel of the test running on a specific environment.

This article explains the factors that define a baseline, and explains the basis on which Eyes which baseline to use for a given test running on a given execution environment. Eyes offers two different policies for deciding which baseline should be used, and these are described below.

The article is divided into two sections:

  • Overview: Intended for new users, this section explains what an Eyes baseline is; which attributes identify the test and execution environment; and when baselines are created, used, and updated.
  • How Eyes selects an appropriate baseline: Intended for advanced users, this section explains the how Eyes chooses which baseline to use when a test has multiple baselines for multiple environments.

Overview

What is a baseline?

A baseline stores a sequence of reference images. When you run an Eyes visual test, you drive the application through a series of application states, and for each state you execute a checkpoint which captures an image of the application in that state. Eyes compares the sequence of captured images to the sequence of reference images stored in the baseline of that test and reports any significant differences.

The images in the baseline depend on the test being run, but they may also depend on other factors, such as the operating system, the type of browser and the browser viewport size. These factors are collectively termed the execution environment.

Eyes automatically maintains multiple baselines for each test, where each baseline stores a sequence of reference images for a specific execution environment.

In summary, a baseline stores a sequence of images that define the expected outcome of capturing checkpoint images of a given test running under a given execution environment.

The life cycle of a baseline

The first time you run a test in a new execution environment, a baseline for that environment won't exist, so Eyes will declare the test to be a new test, and, by default, will create a new baseline that contains the sequence of images captured by the test's checkpoints. Along with the images, Eyes stores information such as the test name and the execution environment.

On subsequent runs of the test in that environment, Eyes will find the existing baseline for that execution environment and will compare the test run checkpoint images to the images in that baseline.

Once a baseline exists and is used as a reference in a test, if you use the Test manager to make changes to the baseline, such as accepting a new or changed step, and then save the baseline, then a new baseline is created - and this baseline will be used as a reference in future tests.

Test and execution environment attributes

Eyes uses the following test and execution environment attributes to identify a particular baseline:

  • testName and appName: Two user-defined strings that describe the tested application and the specific test. Together, they uniquely identity the test.
  • Viewport size: For applications running on a web browser, Viewport size is the dimensions of the browser pane that displays the content. For mobile devices, the Viewport size is the dimensions of the screen.
  • Host environment: This consists of two attributes called the HostOS and the HostApp. HostApp represents that application platform whose screenshots are being captured and checked, so for web applications, this is the browser. HostOS represents the operating system on which the HostApp ran.
  • Version information: Both the HostOS and the HostApp can have version information. One of the difference between the two baseline management models described below is the way they relate to the version information when managing baselines.

Where does Eyes get the test baseline attributes from?

Some of the test attributes are derived by Eyes when you start a test by calling class$eyes$open:

eyes.open(driver, appName, testName, viewportSize)
  • The test is identified by the strings appName and testName.
  • When the images are being captured on the test browser (as opposed to a test that uses the Ultrafast Grid):
    • The Viewport size attribute is defined by the viewportSize parameter.
    • The HostApp attribute is derived from the test browser name and version (i.e. from the driver parameter).
    • The HostOS attribute is derived from the operating system the browser runs on.
  • When the test uses the Ultrafast Grid, the execution environment attributes are derived from the Ultrafast Grid parameters passed to to the Ultrafast Grid target configuration methods: 
     In this case, the Viewport size is based on the
				 viewportwidth
					and
				 viewportheight
				    parameters, the HostApp by the 
				 browsertype,
					 devicename or 
					 iosdevicename
					values passed, and the HostOS is mapped by Eyes internally to the OS used when rendering for that particular browser or device.
    • 
                                

                                
The  class$configuration$sethostapp and  configuration$sethostos methods allow you  to define the execution environment explicitly. A detailed discussion is not in the scope of this article.

                                
How Eyes selects an appropriate baseline

                                
Eyes supports two baseline management policies. These differ from each other in the following main considerations :

                                
    
                                    
  • 
                                        
    In what cases does Eyes support multiple, independent baselines for different HostOS or HostApp versions.
    
                                    
    • 
                                    
  • 
                                        
    In what case will Eyes use an existing baseline for a test even though the HostOS or HostApp version of the baseline is different from the version of the execution environment.
    
                                    
    • 
                                    
  • 
                                        
    When a new baseline is created because there is no existing matching, compatible baseline - can it inherit the checkpoints from an existing related baseline.
    
                                    
    • 
                                

                                
 We will explain each of the policies in detail below, but in a nutshell, the standard policy considers the HostOS version to be part of the baseline signature, and ignores the HostApp major version it doesn't support baseline inheritance. The advanced policy is more flexible for both  HostOS and HostApp versions. In both cases it allows cross version testing as well as supporting independent baselines for different versions. The advanced baseline management policy also allows a newly created baseline to obtain its starting images from a previous version of the baseline.

                                
The standard  baseline management policy

                                
The standard Eyes baseline management policy  enforces compatibility of HostOS major version numbers and disregards the HostApp (Browser) version. For example, a test run on Windows 10 can't use a baseline created on Windows 7, but a test run on Chrome version 80 can use a baseline created on Chrome 72. There will be one baseline for Windows 10 tests and another for Windows 7 tests, but for a given HostOS version, the same baseline is used for all HostApp versions.

                                
This policy is based on the observation that browser versions change frequently and are more likely to be backwards compatible, so reuse of a baseline is appropriate; whereas major HostOS version are infrequent and are more likely to be incompatible, so reuse of a baseline is more likely to be inappropriate.

                                
Sometimes the OS used by the Ultrafast Grid for a browser changes from one browser version to the next. For example Safari version 13 runs on macOS 10 and Safari 14 runs on macOS 11. Although the standard baseline management uses the same baseline for all browser versions, in this particular case, since the HostOS version is different, and this implies a different baseline, a new baseline will be created when the Safari 14 tests first runs. This is one of the use cases where the advanced version baseline selection policy can be useful since it allows baseline inheritance across major OS versions changes, so Safari 14 tests will have an independent baseline - but will inherit their starting baseline from the Safari 13/macOS 10 baseline.

                                
The advanced version baseline management policy

                                
Eyes chooses a baseline based on three criteria: 

                                
    
                                    
  • Finding one or more matching  baselines.
    • 
                                    
  • Selecting baselines from the set of matching baselines that are compatible.
    • 
                                    
  • Selecting the most appropriate  baseline from the compatible baselines .
    • 
                                

                                
The first criteria selects all the baselines associated with the current test that match the test execution environment (i.e. have identical HostOS and  HostApp names and an identical Viewport size).

                                
The second and third criteria deal with a situation where a test has multiple matching baselines with different HostOS and HostApp versions. Eyes needs to select the baseline with HostOS and HostApp versions that are the most appropriate for the corresponding versions of the test execution environment.

                                
These criteria are described in detail in the sections that follow.

                                
Finding a matching baseline

                                
When you start a test,  Eyes searches to see if a baseline exists for your test:

                                
    
                                    
  • Eyes searches for one or more matching baselines (i.e. baselines that have the same Test name, App name, Viewport size, and HostApp and HostOS names). The matching baselines will have different values for the HostOS version and/or HostApp version, based on the HostOS and HostApp versions used when the test was run and the baseline was created.
    • 
                                    
  • If no matching baseline exists, then this is a new test, and, by default, a new baseline will be created based on the checkpoint images captured by the test. The new baseline will be associated with the baseline attributes from the test and its execution environment. 
    • 
                                    
  • If exactly one matching baseline is found,  Eyes will use that baseline. 
    • 
                                    
  • It more than one baseline is found,  Eyes will choose the most appropriate baseline, as explained in the next section.
    • 
                                

                                
Choosing a baseline from multiple versions

                                
If Eyes finds multiple matching baselines, it will look for the most suitable baseline by comparing the HostApp and HostOS version information of the baseline and the execution environment. This version comparison is based on two criteria:

                                
    
                                    
  • The  baseline version must be compatible with the test version. For example, all the major number versions of Microsoft Internet Explorer (e.g. IE 9, IE 10, IE11) are considered to be incompatible with each other. But for a given major version, the minor versions are considered compatible. On the other hand, Eyes considers all versions of Chrome to be compatible. Compatibility is defined fully in the When are versions compatible? section below.
    • 
                                    
  • Versions have an inherent ordering. For example, Eyes will allow a test running on a newer version of a browser to run against a baseline created from an older version as long as it is compatible, but if the test is running on an older version of the browser, it won't use a baseline created  from a test running on a new version.
    • 
                                

                                
Given a choice of several matching, compatible baselines, Eyes will, in general, choose a baseline with the same version as the execution environment. As a second option, Eyes will choose the latest version of a baseline as long as that version is previous (i.e. earlier) to the execution environment version. The following table defines the exact order of priority:

                                

                                    
                                    
                                    
                                    
                                        

                                            Priority
                                            HostApp version
                                            HostOS version
                                        

                                        

                                            1
                                            Same version
                                            Same version
                                        

                                        

                                            2
                                            Same version
                                            Latest previous version
                                        

                                        

                                            3
                                            Latest previous version
                                            Same version
                                        

                                        

                                            4
                                            Latest previous version
                                            Latest version (HostApp latest has priority)
                                        

                                    
                                

                                
If no baseline can be found with the same or earlier HostApp and HostOS versions, then the test is treated as a new test.

                                
When are versions compatible?

                                
The HostOS and HostApp compatibility rules are based on the way versions are defined in browsers. A version string is assumed to be constructed of one or more components separated by periods (e.g. 10.1.3). The leftmost component is called the major version number and must be numeric. The other components may be numeric or alphanumeric (i.e. a-z, A-Z, 0-9, or _). Version Format section below explains the expected format in more detail.

                                
The following table defines compatibility for different values of HostOS and HostApp names:

                                

                                    
                                    
                                    
                                    
                                        

                                            Attribute
                                            
                                            Compatibility
                                            
                                            Example
                                            
                                        

                                        

                                            HostOS = any value
                                            All versions are compatible
                                            Android 10.1 baseline is compatible with both Android 10.0 and Android  9.2 baselines.
                                        

                                        

                                            HostApp =  
“Safari”, “Firefox”, “Edge”, “Chrome”, “Opera"
                                            All versions are compatible
                                            Chrome 81 baseline is compatible with any other version of Chrome.
                                        

                                        

                                            HostApp = “IE”

                                            Versions are compatible if the major component matches
                                            IE 10 baseline is not compatible with IE 9 baseline. Any IE 10 sub version is compatible with any other IE 10 version.
                                        

                                        

                                            
                                                
HostApp = any other value


                                                
NOTE
                                                

                                            		
                                            Versions are compatible if the major component matches
                                            MyApp 1.1 is compatible with MyApp 1.2 but is not compatible with MyApp 2.2.
                                        

                                    
                                

                                
Running cross environment tests

                                
A cross environment test is a test that runs against a baseline with an execution environment that is different from the test execution environment. This allows you, for example, to test your application on a number of different browsers against  a single baseline that has been verified on another browser.

                                
You run a cross browser test by calling the  class$configuration$setbaselineenvname method to set the baseline environment name you want to test against. The HostOS, HostApp and Viewport size associated with this name is defined using the Test manager. This process is described in detail in the article Running cross-environment tests.

                                
When you run a cross environment test, the rules for selecting the appropriate version of the baseline are different from a test where the baseline is based on the test execution environment:

                                
    
                                    
  1. If the test execution environment does not match the  environment defined by the baseline environment name, then the latest compatible version of the OS and browser will be used. This version may be previous to, or later than the execution environment version as long as it is compatible.
    2. 
                                    
  2. If the test execution environment matches the  environment defined by the baseline environment name, then the baseline is chosen based on the version information of the test environment, as described in Choosing a baseline from multiple versions above.
    3. 
                                

                                
Version Format

                                
 Eyes automatically extracts the version string from the execution environment information. If you define the HostOS or HostApp yourself (i.e. you use the methods  configuration$sethostapp or  configuration$sethostos) and you want Eyes to look for a fallback baseline, then you need to format the version string to meet the following guidelines:

                                
    
                                    
  • The version string must appear after the name separated by exactly one space. The version number may not be followed by any non white space characters.
    • 
                                    
  • The version string must be a sequence of one or more components separated by a period (e.g. 3.0.23a.t3), where the leftmost component  (the major component) must be numeric, and the other components can be numeric or alphanumeric (e.g. a-z,A-Z,0-9, or _). 
    • 
                                    
  • When comparing two component strings to see which is the latest, the strings are compared component by component from the left most (major) component to the right. If both components are numeric, they are compared numerically, if they include non numeric characters, they are compared alphabetically. If the number of components is not identical (e.g. 1.0.0 and 1.0.0.A) but the component of the shorter version string matches that of the longer version string, the shorter version string is considered "older" (so 1.0.0 is considered older than 1.0.0.A).
    • 
                                    
  • If the version string does not match the above format, then it is considered part of the name. This means that a change in version information will create a new baseline, with no fallback to a previous compatible version.
    • 
                                

                                
For further information, see our video, Validating Baselines in Applitools Eyes and Creating Multi-Basleine Variations: