Batching tests in a distributed environment
Grouping tests into batches, allows you to see and manage the results of multiple related tests together in the Test Manager. The article Grouping tests into batches with the SDK explains how to batch tests when they run in the same test execution.
This article, explains how to batch results of tests that are executed in different processes or on different machines. This technique is useful if, for example, your tests are coded in different programming languages, your tests need to run on multiple target platforms (e.g. Linux, Windows, Mac, and mobile devices), or you run tests concurrently on multiple CPUs or distributed processors. The techniques described here are also useful when you run a single process test using multiple runners.
To batch together tests that run in separate processes or separate runner, you need to do three things:
- Prevent auto-closure of batches by runners
- Associate tests with a common batch
- Close the shared batch
Avoid batching together unrelated tests unnecessarily, since this may increase the number of tests in a batch to a large number and can cause severe performance issues and/or failures.
Prevent auto-closure of batches by runners
If you use a ClassicRunner
or VisualGridRunner
in your tests then Implicit batching by a runner, and by default, the runner closes the batch(es) in the call to class$visualgridrunner$getalltestresults
after the tests have completed.
If you are using multiple executions, or more than one runner in a single execution, and you want the tests to be in a common batch then you have to disable the auto closing of batches. There are two ways to do this:
-
By calling the method
class$visualgridrunner$setdontclosebatches
on every runner immediately after it is created, as shown below: -
By defining the environment variable
APPLITOOLS_DONT_CLOSE_BATCHES
before the test runs.The same methods are available in the
ClassicRunner
class and theVisualGridRunner
class.
Associate tests with a common batch
When you run test in a single process, you can batch them by assigning a shared BatchInfo
object to every Eyes instance using one of the methods described in How to group tests into batches.
When tests are run in separate runs, you cannot share a BatchInfo
object. Instead, each test creates its own BatchInfo
object, and the tests are batched by explicitly assigning the BatchInfo
object a common Batch ID.
We recommend that you use the same batch name for all the tests. However, even if you use different names, it is the common batch ID that groups the tests together. The batch name displayed in the Test Manager is the batch name of the first test that runs.
Typically sharing a batch id is done using the following steps:
-
Generate a unique ID string, for example by using date and time stamp and some machine ID. Since batches are stored on the server for a long time, you need to ensure that the ID will be unique in the long term.
-
Distribute the unique ID to all your test programs (e.g. by writing it to a network file or a database).
-
In each test program, obtain the shared ID and assign it to its
BatchInfo
object. This can be done in two ways: -
By obtaining the ID in a script and setting the environment variable
APPLITOOLS_BATCH_ID
to the unique ID before starting the test process. The SDK will use this value of this environment variable if it is defined and if the batch ID is not set explicitly. -
By obtaining the batch ID in your code and setting the
BatchInfo
object explicitly as illustrated below:
The call to the function myGetUniqueBatchID()
represents the code you need to add that obtains the ID from wherever it is stored.
Close the shared batch
Closing a batch means that:
-
In the Test Manager, the status of the batch will change from Running, Test and batch status of the tests in the batch.
-
If Batch completion notification has been enabled, a notification will be sent.
You can close a batch by using the BatchInfo
class in an SDK that supports it (e.g. Java) or you can send an HTTP request directly to the Eyes server using a program such as curl.
Closing a batch with the BatchClose class
The following snippet shows how to close a batch using the
Note that before calling the batchclose$close
method, you need to pass one or more batch ids to the method batchclose$setbatchid
. In this example we use the value of the environment variable APPLITOOLS_BATCH_ID, typically you would use the same method you used to assign the ID when defining the batch as described in the previous section Associate tests with a common batch.
Sending the server an HTTP requests directly
For SDKs that don't support the BatchInfo
class, you can close the batch using curl
as illustrated in the following Bash script:
For Linux:
export APPLITOOLS_SERVER_URL=https://eyesapi.applitools.com
curl -v -X DELETE "$APPLITOOLS_SERVER_URL/api/sessions/batches/$APPLITOOLS_BATCH_ID/close/bypointerid?apiKey=$APPLITOOLS_API_KEY"
For Windows:
set APPLITOOLS_SERVER_URL=https://eyesapi.applitools.com
curl -v -X DELETE "%APPLITOOLS_SERVER_URL%/api/sessions/batches/%APPLITOOLS_BATCH_ID%/close/bypointerid?apiKey=%APPLITOOLS_API_KEY%"
The script assumes that the variables APPLITOOLS_SERVER_URL
, APPLITOOLS_API_KEY
and APPLITOOLS_BATCH_ID
have the appropriate values for your environment.
If you use the public cloud, the value of APPLITOOLS_SERVER_URL
is 'https://eyesapi.applitools.com'. If you run the Eyes server on a private cloud or on-premise system, then change the value assigned to APPLITOOLS_SERVER_URL
to the URL of your Eyes server.