Grouping tests into batches with the SDK

In the Test Manager, test results are grouped into batches. By grouping related tests into batches you can view, process and manage the test results as a unit. For a broad view of all the features associated with batches see the article How to organize your tests with batches. This article focuses on how to use the SDK to assign tests to specific batches.

Batch essentials

Before diving into the details of how to assign tests to batches, it is useful to understand some basic concepts.

Default and Explicit Batching

Test results are always associated with a batch, the batch may be the default batch created by the SDK or a batch that you create, name and assign to tests explicitly. The default batching scheme depends on which SDK you use:

  • In SDKs that implement the ClassicRunner and/or VisualGridRunner, all the tests that run on a given runner are by default associated with a batch created by the SDK.

  • In SDKs that do not implement runners (such as the Image SDKs), by default each test runs as a separate batch created by the SDK.

In most cases, it is advisable to use explicit batching in order to give the batch a descriptive name. Other things that you can achieve by using explicit batching are:

  • Batch tests from multiple runners into a single batch.

  • Batch a number of related tests into a single batch when using a non runner SDK.

  • Batch a number of related tests that are running in separate test executions.

Batch name and Batch ids

It is important to understand the difference between the batch name and the batch id.

The batch name is the user visible name displayed in the list of batches in the Test Manager. The batch name is not unique, every time you run a batch it will have the same name. It is possible to give two distinct batches (i.e. batches that include different tests) the same name, although in most cases this is not recommended practice.

The batch id is a unique identifier of a specific batch run. If you run a batch of tests a number of times they will all have the same name, but each will have a unique id. By default, the SDK assigns each batch a unique ID. The SDK provides you with a way to associate a specific batch ID with a batch, and this allows you to batch tests that run in different processes or on different hardware.

How to group tests into batches

You can assign tests to a common batch in any of the following ways:

Each of these methods is described in detail in the sections that follow.

Implicit batching by a runner

By default, all tests that are run using an SDK that supports the ClassicRunner or VisualGridRunner are batched into a single batch whose name is the name of the first test to run. You can use the methods described below to give an appropriate name to the batch and to implement any batching scheme you want, for example, to separate unrelated tests run by a runner into different batches, or to combine related tests from multiple runners into a single batch.

Explicit batching with a shared BatchInfo object

The methods describe below use a shared BatchInfo object to assign tests to a batch represented by that object. This is convenient when the tests run in the same process. Two methods are described - using a shared Configuration object and directly assigning each test to a batch.

Explicit batching using a Configuration object

If you are using a shared Configuration object to set common properties for all tests, then you can share the BatchInfo object as well by using the method configuration.setbatch:


                                
Note that the call to Eyes.setconfiguration should be made before any other test specific configuration, since it assigns all the Configuration properties - either to the values set explicitly or to the default value.

                                
Explicit batching the Eyes object

                                
You can also associate a test with a batch by assigning the global, shared instance of the  BatchInfo class to the Eyes instance of the test using the method  eyes.setbatch:

                                
If you previously set a Configuration object on this Eyes instance then the call to  eyes.setbatch will override it. 

                                
Explicit batching with a shared batch Id

                                
If the tests are running on different processes, or on different machines, then instead of sharing a BatchInfo instance, each process creates a BatchInfo object, and initializes it with a shared, common batch Id. The snippet below shows the basic technique. The ID can be shared in any convenient way, symbolized here by the call to the method getSharedId which you need to implement to obtain the ID using whatever method you choose.

                                
Of course, if you are not using a shared Configuration object on all the tests you want to batch then you can also use the eyes$setbatch$morp to assign the BatchInfo instance to a specific test.

                                
For more information about batching tests that are run in separate processes see  Batching tests in a distributed environment

                                
Batch completion termination

                                

As explained in the article Batch completion notifications you can set up Eyes using the Eyes Test Manager so that when a batch completes and it includes tests for a defined application users will get an email or a Slack notification. In order for this to happen, you need to enable this feature when the BatchInfo object is created, for more information see Setting up batch completion notifications in the SDK.