In our previous article, we learned what autonomous testing is all about: autonomous testing is when a tool can learn an app’s behaviors and automatically execute tests against them. It then provides results to humans who can determine what’s good and what’s bad. Fully autonomous testing solutions are not yet available today, but we can get part of the way there with readily available tools. Let’s learn how to build our own semi-autonomous solution using Playwright and Applitools Eyes!

The solution sketch

Let’s say you have a website with multiple pages. It would be really nice if a test suite could visit each page and make sure it looks okay: no missing buttons, no overlapping text, and no other kinds of visual bugs. The tests wouldn’t be very sophisticated, but they’d quickly catch a lot of problems. They’d be like smoke tests.

There’s a straightforward way to do this. Most websites have a sitemap file that lists the links for all the pages. We could use a browser automation tool like Selenium, Cypress, or Playwright to visit each of those pages, and we could use a tool like Applitools Eyes to capture visual snapshots of each page. Every time we run the suite, it would automatically discover new pages and avoid removed pages. Existing pages would be checked for visual differences. We wouldn’t need to explicitly code what to check – the snapshots would implicitly check everything on the pages! With the Applitools Ultrafast Grid, we could even test these pages against different browsers, devices, and viewport sizes.

This kind of test suite is technically “autonomous” because we, as human testers, don’t need to explicitly write the tests. The sitemap provides the pages, and visual testing covers the assertions.

The website to test

Last year at Applitools, we replaced our old tutorial website with a new website that uses Docusaurus, a very popular documentation framework based on React. We also rewrote several of the guides for our most popular SDKs. Presently, the site has about 60 pages of varying length. Since many of the pages host similar content, we use components to avoid duplication in text and in code. However, that means any change could inadvertently break multiple pages.
Our tutorial site is currently hosted at https://applitools.com/tutorials/:

The tutorial site also has a sitemap file at https://applitools.com/tutorials/sitemap.xml:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:news="http://www.google.com/schemas/sitemap-news/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1" xmlns:video="http://www.google.com/schemas/sitemap-video/1.1">
    <url>
        <loc>https://applitools.com/tutorials/</loc>
        <changefreq>monthly</changefreq>
        <priority>0.5</priority>
    </url>

    <url>
        <loc>https://applitools.com/tutorials/guides/advanced-tricks/troubleshooting-issues</loc>
        <changefreq>monthly</changefreq>
        <priority>0.5</priority>
    </url>
...

It would be very helpful to test this tutorial site with the semi-autonomous testing tool we just sketched out.

Writing the autonomous testing code

To follow along with this article, you can find the GitHub repository for the project at https://github.com/AutomationPanda/auto-website-testing. In the package.JSON file, you’ll find all the packages needed for this project:

• Playwright
• Applitools Eyes SDK for Playwright
• ts-node

Since I’ll be developing my code in TypeScript instead of raw JavaScript, the ts-node package will allow me to run TypeScript files directly.

{
  "name": "auto-website-testing",
  "version": "1.0.0",
  "description": "A semi-autonomous testing project that visually tests all the pages in a website's sitemap",
  "main": "index.js",
  "scripts": {},
  "repository": {
     "type": "git",
     "url": "git+https://github.com/AutomationPanda/auto-website-testing.git"
   },
   "keywords": [],
   "author": "",
   "license": "ISC",
   "bugs": {
      "url": "https://github.com/AutomationPanda/auto-website-testing/issues"
   },
   "homepage": "https://github.com/AutomationPanda/auto-website-testing#readme",
   "devDependencies": {
     "@applitools/eyes-playwright": "^1.13.0",
     "@playwright/test": "^1.29.1",
     "ts-node": "^10.9.1"
   }
}

The main file in our project is autonomous.ts. This isn’t a typical Playwright test that has described blocks and test functions, but rather autonomous.ts is just a plain old script. The first thing we need to do is read in some environment variables. For test inputs, we’ll need the base URL of the target website. We’ll need the site name of the website for logging purposes and reporting purposes, and we’ll also need a level of test concurrency for the Applitools Ultrafast Grid, which will handle our visual snapshots. If you’re on a free Applitools account, you’ll be limited to one, but I’ll be using a bit more in this example.

import { chromium } from '@playwright/test';
import { BatchInfo, Configuration, VisualGridRunner, BrowserType, Eyes, Target } from '@applitools/eyes-playwright';

(async () => {

     // Read environment variables
     const BASE_URL = process.env.BASE_URL;
     const SITE_NAME = process.env.SITE_NAME;
     const TEST_CONCURRENCY = Number(process.env.TEST_CONCURRENCY) || 1;

Once we read those in, we want to validate those environment variables to make sure their values are given and they’re good. Then what we’ll do is we will figure out what the sitemap URL is. Basically, the sitemap URL will be the base URL plus this standard XML file name.

 // Validate environment variables
 if (!BASE_URL) {
    throw new Error('ERROR: BASE_URL environment variable is not defined');
 }
 if (!SITE_NAME) {
    throw new Error('ERROR: SITE_NAME environment variable is not defined');
 }

 // Parse the base and sitemap URLs
 const baseUrl = BASE_URL.replace(/\/+$/, '');
 const sitemapUrl = baseUrl + '/sitemap.xml';

Next, we will set up Applitools to be able to do visual testing. These are all fairly standard Applitools SDK objects. If you take one of our Applitools SDK tutorials, you’ll see things just like this. Basically, we’ll need a visual grid runner to connect to the Ultrafast Grid for rendering our snapshots. We’ll create a batch which will have the name of our site so that we can see reporting, and we’ll have a configuration object to specify things like the batch, the browsers, and all these other things we want.

 // Create Applitools objects

 let runner = new VisualGridRunner({ testConcurrency: TEST_CONCURRENCY });
 let batch = new BatchInfo({name: SITE_NAME});
 let config = new Configuration();
 let widthAndHeight = {width: 1600, height: 1200};
 let snapshotPromises: Promise<any>[] = [];

With the configuration, we’re going to set the batch and we’re going to add one browser to test. I want to test Chrome with this particular viewport size. If we wanted to, we could also test other browsers in the Ultrafast Grid such as Firefox, Safari, and Edge Chromium. Even if you don’t have those browsers installed on your local machine, it’s all going to be done in the Applitools cloud. For this example, we’ll use one browser.

 // Set Applitools configuration
 config.setBatch(batch);
 config.addBrowser(1600, 1200, BrowserType.CHROME);
 // config.addBrowser(1600, 1200, BrowserType.FIREFOX);
 // config.addBrowser(1600, 1200, BrowserType.SAFARI);
 // config.addBrowser(1600, 1200, BrowserType.EDGE_CHROMIUM);

Now comes the fun part of getting that sitemap file. What we’ll need to do is launch a browser through Playwright, we’ll just use Chromium. Then we’ll need to create a new browser context from that browser and we’ll give it a standard width and height viewport. Then we’ll get a page object from that context, because with Playwright, all interactions happen through a page object.

 // Set up a browser
 const browser = await chromium.launch();

 // Set up a sitemap context and page
 const sitemapContext = await browser.newContext({viewport: widthAndHeight});
 const sitemapPage = await sitemapContext.newPage();

Once we’ve got that page, now we can visit the sitemap page and we can try to find all of the page links inside that sitemap file. Even though it’s XML, Playwright can still parse it just like it’s a regular webpage. Once we’ve got that list of page links, then we’re going to close that session and so this Playwright session will be done.

 // Get the sitemap
 await sitemapPage.goto(sitemapUrl);
 const pageLinks = await sitemapPage.locator("loc").allTextContents();
 sitemapContext.close();

Now that we have the list of all pages from the sitemap, we can visit each one and capture a snapshot. To do that, we’re going to iterate over that list with a for loop for each page we visit. We’re going to make a promise so that we can capture those snapshots asynchronously. In the background, for each page, we are going to create a new browser context and then from that context, create a new page object, and then start an Applitools Eyes session. This is what enables us to capture those visual snapshots.

 // Capture a snapshot for each page
 for (const link of pageLinks) {
     snapshotPromises.push((async () => {

       // Open a new page
       const linkContext = await browser.newContext({viewport: widthAndHeight});
       const linkPage = await linkContext.newPage();

       // Open Eyes
       const eyes = new Eyes(runner, config);
       await eyes.open(
           linkPage,
           SITE_NAME,
           link.replace(baseUrl, ''),
           widthAndHeight
     );

Taking the snapshot is pretty basic. We just visit the page and we say eyes.check, and we take a picture of the whole window. Once we do that, we can close our session so that Applitools knows that’s the only snapshot we’re taking. Firing these off in different promises means that we can run them asynchronously in the background, letting Applitools Eyes crunch through all of the visual validations, and that way we’re not waiting for them one at a time. They’ll just all go. Once we fired off all of the visual snapshots, we can wait for those promises to join at the end.

       // Take the snapshot
       await linkPage.goto(link);
       await eyes.check(link, Target.window().fully());
       console.log(`Checked ${link}`);

       // Close Eyes
       await eyes.close(false);
       console.log(`Closed ${link}`);

    })());
 }

And finally, once that’s complete, we can close the browser and be done with testing.

 // Close all Eyes
 console.log('Waiting for all snapshots to complete...');
 await Promise.all(snapshotPromises);

 // Close the browser
 await browser.close();
 console.log('Complete!');

})();

That’s all there is to our autonomous testing script. It’s pretty concise – only about 80 lines long. It doesn’t take a whole lot of logic to write autonomous tests.

Running the tests

Let’s run the script to test it out. The website I’m going to use in this example is the Applitools tutorial site, which has about 60 pages. You can use any site you want as long as you set your environment variables. If you want to run this, you will need an Applitools account, which you can register for free with your email or GitHub account. Once you register an account, you’ll take your Applitools API key and set that as an environment variable.

We need to run our tests in the terminal. If this is your first time, you’ll need to run npm install to install the npm packages as well as npx playwright install to install the Playwright browsers. Once you’ve run those installers and set your environment variables, you just need to run npx ts-node autonomous.ts to run the script.

The script fetches the sitemap file, parsing out all of those links, and it’s firing off promises to start capturing visual snapshots for each one that’s happening asynchronously. The messages saying “Checked” link means those have now been initiated. What we’ll start to see is as they complete one by one in the Applitools Ultrafast Grid, we’ll see how the session becomes closed. Closed images code, JavaScript, closed mobile browser, that means one by one, the visual testing has been completed.

While the test is running, we can see results in the Applitools Eyes dashboard as a batch.

If I look at all the tests they’re popping in, we can see some of them are still running while others have passed.

If I want to see what the visual comparisons look like, I can compare them side by side. If there are no visual differences, the test for this page will pass.

You can see on the page it captures everything, the title bar, the sidebar table of contents, main body and footer. It will even scroll all the way to the bottom to make sure it gets the full page worth of contents.

No matter how long the page is, everything is being compared visually. It’s pretty cool to see just how many tests our autonomous testing solution can uncover. As I said before, the tutorial site right now has about 60 pages, which means that’s 60 tests that I didn’t have to explicitly write. Playwright plus Applitools Eyes took care of it all for me.

Improving the tool

This example of a semi-autonomous testing tool is rather basic. There are plenty of ways we could improve it:

• Decoupling our tests: We could write separate scripts for fetching sitemap links and taking the visual snapshots. That would let us provide links to visit by ways other than a sitemap file. We could also add an intermittent step to filter links from a sitemap file.
• Expanding test coverage: We could add settings to test different browsers, devices, and viewports. This would provide autonomous cross-browser and cross-device testing coverage for multiple screen sizes.
• Target specific regions: We could exclude parts of pages like navigation bars and sidebars. This could allow us to target specific portions of a page in our tests while ignoring content that is dynamic or that we may want to test separately.

Even with the basics before any of these suggestions are implemented, this tool still provides a lot of value for a little bit of work. You can clone it from the GitHub repository and try it yourself. Let us know @Applitools how you use it!
Read more about Applitools Eyes.

Applitools is working on a fully autonomous testing solution that will be available soon. Reach out to our team to see a demo and learn more!

Salesforce is not just one of the most popular CRM platforms; it’s an ecosystem that combines almost anything that a business needs under one roof. Gone are the days when entrepreneurs were willing to use hundreds of different tools through separate services. Nowadays, companies want everything they need in one product, under one domain – think messenger, documentation, product site, analytics, and more. That’s what Salesforce provides to its customers – a single place where businesses can conduct their work.

Since its founding in 1999, Salesforce has undergone many changes. While it’s natural for a product to evolve to meet its customers’ demands, in 2018, Salesforce introduced its Lightning Platform, which was a complete overhaul of the entire user experience at Salesforce.

With a highly dynamic platform like Salesforce, developers who create apps and integrations on Salesforce need a reliable way to test their applications. Test automation and, more specifically, visual testing ensures a scalable way of validating core functionality that your customers rely on.

Why Testing Matters

As with any product, we want to be confident that our application doesn’t break overnight. We need to ensure that the behavior that our customers are used to stays the same unless we intentionally want to introduce a change. This reason is why end-to-end testing is vital since it covers the core user workflows like logging in, creating an account, buying an item, and many other use cases.

While other levels of testing, like unit or integration tests, are fantastic for ensuring a short feedback loop for our developers, end-to-end tests give us the highest level of confidence that the overall product continues to work well for our customers.

Suppose we are working within the Salesforce ecosystem. In that case, we need a reliable way to test our Salesforce product to ensure that our Salesforce integration or a custom app is fully functional. 

This post looks at how TestZeus, a test automation library for Salesforce apps, and Applitools Visual AI can provide us with an optimized, efficient, and deterministic way to automate our customer scenarios by writing less code yet increasing test coverage.

Introducing TestZeus for More Effective Test Automation for Salesforce

TestZeus is an open-source UI automation framework that provides an efficient way to find elements on Salesforce apps. It relies on what’s called AutoLocators strategy that uses the Salesforce Developer API to find UI elements on the webpage. 

In order for us to properly demonstrate how TestZeus works, we first need to set up our Salesforce environment.

Setting up Salesforce Environment

Salesforce is an entire ecosystem created around its Customer Relationship Management (CRM) system. And as fun as testing in production could be, we should make a development environment with Salesforce.

Luckily, Salesforce has a highly engaging community of developers and many learning resources. One of the community members has covered in great detail how to setup what’s called a Salesforce Developer Edition instance. In this article, we will only focus on the core parts of the process (for more information, you can read the community forum).

After you navigate to the signup page, you will need to fill out some basic information about your account:

Signup Page for Salesforce Developer Edition

Immediately after you sign up, you will receive a confirmation email that will look something like this:

Confirmation Email Post Instance Creation

Apart from pressing “Verify Account” button, you will need to make sure to capture your instance’s URL (HOME_URL as we will call it in the test properties) and username (USERNAME in the test properties):

Instance URL and Username Properties

When you first attempt to log into the Salesforce instance, it will prompt you to create a password which we will later use as the PASSWORD property in the tests.

New Password Creation Flow

Using Salesforce Developer API

TestZeus requires us to use the Salesforce Developers API. For this article, we will need a security token for your account and a consumer key/secret combination.

Without diving into too many details, a security token is another way to authenticate the Salesforce Developer API. At the same time, a consumer key/secret combination lets Salesforce control the scope of your API calls.

Getting a security token

The security token is one of the components required for API authentication. To get it for the first time, you need to “Reset Security Token” through the Salesforce UI. This video goes through details on how to do it step-by-step. Ultimately, you will receive an email containing a security token that we will use as SECURITY_TOKEN properties in our tests.

Security Token for Salesforce Developer API

Acquiring a consumer key/secret combination

The last piece of information we need to authenticate your tests successfully is what’s called a Consumer Key and Secret. We will need to create a Connected app that will, in turn, provide the authentication details. You can think of this step as a way to control the permission scope of the API usage.
This video walks through the steps to create the Connected app and retrieve its authentication details. In short, we need first need to create a New Connected App:

Salesforce App Manager

Then fill in all the required fields, enable OAuth settings and choose the necessary scope (as shown in the video):

Creating a Connected App

Following the app’s creation, you must click “Manage Consumer Details” to retrieve the authentication information:

Connected App Details

This new window will show us the exact data we need: a Consumer Key and a Consumer Secret:

Consumer Details

In our test, we will later use this information as CONSUMER_KEY and CONSUMER_SECRET properties.

Functional Testing with TestZeus

Now with all the necessary information and demo environment set up, we can proceed with writing our tests. For this article, we put together a small project that you can see here. We will first walk through this code to show how to use a TestZeus framework for testing a Salesforce application and, later, add a visual testing component using Applitools.

Preparing Test Scenario

For our article, we will do a primary case of validating UI for a test account we create beforehand.

In your Salesforce instance, navigate to Accounts and click “New account”:

Accounts Tab

Then we will create an Account named “Test Account,” and our test scenario is ready to be used.

Creating New Account

Setting Up TestZeus

First, we need to clone our demo project:

git clone git@github.com:dmitryvinn/applitools-testzeus-demo.git

Our demo project uses Java programming language and Maven as its build system. Hence, we encourage you to use an IDE like Intellij IDEA to simplify the development experience.

After you open the project with your IDE of choice, you must use Maven to build the project and download all necessary dependencies. One of the critical dependencies we are relying on for this part of the article is TestZeus dependency:

<dependency>
            <groupId>com.testzeus</groupId>
            <artifactId>Test_Zeus</artifactId>
            <version>1.0.2</version>
        </dependency>

The next step is to fill out all the required properties we will use in our tests in the config.properties file. So far, we have covered all but the APPLITOOL_KEY property, which we will discuss later.

Writing Our First TestZeus Test

In this article, we explore a simple use case of validating an account we created beforehand.

For brevity purposes, we will skip going into the general Selenium Webdriver code and focus only on specific details around TestZeus.

First, we need to authenticate our test, so we can use the Salesforce Developer API to work with data objects, retrieve specific selectors, and more.

In BaseTest.java, we have the following method we invoke in our setup() to handle authentication:

private void authenticateUser() {
    HTTPClientWrapper.SFLogin_API(HOME_URL, TOKEN_GRANT_URL, CONSUMER_KEY, CONSUMER_SECRET, USERNAME, PASSWORD + SECURITY_TOKEN);
}

Then, we need to create a new test named AccountsTest.java where we will have our #findTestAccount() test case.

We will begin our test scenario by Arranging it, or in other words, by setting up our test case.

We will first initiate an SFPageBase with the current webdriver instance and then navigate to the home URL:

final SFPageBase salesforcePage = new SFPageBase(driver);

salesforcePage.openHomepage(HOME_URL);
salesforcePage.maximize();

Now, we will proceed to log into our demo environment:

driver.findElement(By.id("username")).sendKeys(USERNAME);
driver.findElement(By.id("password")).sendKeys(PASSWORD);

final WebElement loginButton = driver.findElement(By.id("Login"));
salesforcePage.safeClick(loginButton);

In the next step, we will transition into the Act portion of the test case, starting with test-specific actions.

We will use TestZeus’ SFBasePage to navigate to the Accounts tab and open our “Test Account”:

  salesforcePage.appLauncher("Account");

final WebElement testAccountItem = driver.findElement(By.xpath(String.format("//a[@title='%s']", TEST_ACCOUNT_NAME)));
 salesforcePage.safeClick(testAccountItem);

Lastly, in the Assert stage of the test case, we will validate one of the fields shown on the UI:

driver.findElement(By.xpath("//p[text()='Account Owner']"));

As we can see, TestZeus helps simplify waiting for UI elements, navigation, and working with Salesforce Objects.

Now with this highly efficient way of locating UI elements using TestZeus, we can take our test automation to the next level with Applitools and visual testing.

Taking Next Steps: Functional Testing for Salesforce apps with Applitools and TestZeus

Functional testing is a type of testing that focuses on the core functionalities of an application while following predetermined steps. In other words, it’s a way to evaluate a scenario with two potential outcomes, failure or success, using the same input.

While there are many functional testing levels like unit or integration testing, end-to-end tests give us the highest confidence level in customer use cases. It’s also one of the most expensive types of testing since it takes a lot of effort to write assertions for every single element of the app and then keep maintaining this workflow. But not with Applitools and Visual AI analysis.

With the Visual AI in Applitools Eyes, rather than writing assertions focused on just one object, Applitools lets us capture the entire web page or mobile app screen, then send it to the Applitools Visual AI for analysis.

Simplifying Functional Testing with Applitools

This visual testing allows us to see our app from our customers’ points of view. And that’s where the Applitools Visual AI comes in! This AI-based visual testing tool can enhance the existing end-to-end test coverage to ensure our app’s user experience conforms to the expected design.

Using AI algorithms, Applitools allows developers, testers, and product/business teams to effectively compare visual elements, with corresponding baselines, across various screens to find visible defects in your test environments. This way, you can prevent functional and visual defects from escaping to production and affecting your customers negatively.

Applitools has integrations with numerous testing platforms like Cypress, WebdriverIO, Selenium, and many others. This article will showcase Applitools with TestZeus to add visual test coverage to our Salesforce app.

Setting up Applitools

Applitools has fantastic documentation on how to get started with the project. For this article, we will first need to register with Applitools and retrieve an APPLITOOLS_API_KEY for our config.properties.

We will also add the following dependency to our Maven build:

      <dependency>
            <groupId>com.applitools</groupId>
            <artifactId>eyes-selenium-java5</artifactId>
            <version>5.39.0</version>
        </dependency>

Now, we are all set to write our test cases with Applitools.

Visual Testing with Applitools and TestZeus

By using this great starter project, we can quickly enhance our existing test suite with visual test automation using Applitools.

First, we will create a new test class named AccountsApplitoolsTest.java. Then, we will need to add some additional setup code to enable Applitools and its Ultrafast Grid, so we can scale our tests in the future.

    @BeforeAll
    public static void setUpConfigAndRunner() {
        runner = new VisualGridRunner(new RunnerOptions().testConcurrency(1));

        config = new Configuration();
        config.setApiKey(APPLITOOLS_API_KEY);

        final BatchInfo batch = new BatchInfo("Salesforce Accounts Page Tests with TestZeus");
        config.setBatch(batch);

        config.addBrowser(1024, 768, BrowserType.CHROME);
    }

In the above code snippet, we set our tests to be run one at a time (the limit for free Applitools accounts), specified the Applitools API Key, and chose Chrome as our test browser.

The next step is to “open the eyes” which is an API terminology for starting visual testing for our test suite:

    @BeforeEach
    public void openBrowserAndEyes(TestInfo testInfo) {
        eyes = new Eyes(runner);
        eyes.setConfiguration(config);

        eyes.open(
                driver,
                "TestZeus Demo",
                testInfo.getDisplayName());
    }

Now, similarly to the previous example with TestZeus, we will have a new test named findTestAccount(). One of the best things about using Applitools is that we don’t have to make any additional changes to our test scenario! So everything from Arrange to Act steps will remain the same:

    @Test
    public void findTestAccount() throws Exception {

        // Arrange
        final SFPageBase salesforcePage = new SFPageBase(driver);

        salesforcePage.openHomepage(HOME_URL);
        salesforcePage.maximize();

        driver.findElement(By.id("username")).sendKeys(USERNAME);
        driver.findElement(By.id("password")).sendKeys(PASSWORD);

        final WebElement loginButton = driver.findElement(By.id("Login"));
        salesforcePage.safeClick(loginButton);

        // Act
        salesforcePage.appLauncher("Account");

        final WebElement testAccountItem = driver.findElement(By.xpath(String.format("//a[@title='%s']", TEST_ACCOUNT_NAME)));
        salesforcePage.safeClick(testAccountItem);

The only change we will need to make is in the Assert part of the test case. Here, we will have code to validate that the page loaded correctly and that its layout matches between test runs. We could also validate a snapshot of the entire page with Applitools, but Salesforce apps are known to have varying load times. So the layout match level helps to avoid false positives:

        eyes.check(Target.window().fully().withName("Test Account").layout());

We are almost done! To make sure that our Applitools tests are closed correctly after their run, we will need to add this @AfterEach method:

    public void cleanUpTest() {
        eyes.closeAsync();
    }

With this, our first test with Applitools and TestZeus for the Salesforce app is complete. Let’s visit our Applitools Dashboard to review the results:

Visual Test Results on Applitools

Wrapping Up

Salesforce is an ever changing ecosystem of apps and products under one platform. That’s why tools like TestZeus are beneficial in simplifying both the creation and, more critically, maintenance of our tests.

Combined with testing automation platforms like Applitools, we can take our testing infrastructure to the next level by focusing on scale, cross-platform support, and a short feedback loop.

ICYMI: Check out Robin Gupta’s Test Automation University course, Automating Salesforce Tests with TestZeus.

The AI promise

AI is not a new technology. Recently, the field has made huge advancements.

Currently it seems like AI technology is mostly about using ML (machine learning) algorithms to train models with large volumes of data and use these trained computational models to make predictions or generate some outcomes.

From a testing and test automation point-of-view, the question in my mind still remains: Will AI be able to automatically generate and update test cases? Can it find contextual defects in the product? Can it eventually inspect code and the test coverage to prevent defects getting into production?

ICYMI: Read my recent article on this topic, AI: The magical helping hand in testing.

The promising future

Recently I came across a lot of interesting buzz created by ChatGPT, and I had to try it out. 

Given I am a curious tester by heart, I signed up for it on https://chat.openai.com/ and tried to see the kind of responses generated by ChatGPT for a specific use case.

Live demo: Watch me demonstrate how to use ChatGPT for testing and development in the on-demand recording of Unlocking the Power of ChatGPT and AI in Testing: A Real-World Look.

What is the role of AI in testing?

I started with a simple question to ChatGPT. The answer was interesting. But the answer was nothing different than what I already knew.

So I thought of asking some more specific questions.

Create a test strategy using AI

I asked ChatGPT to create a test strategy for testing and automating Amazon India shopping app and website.

I was blown away by the first response. Though I can’t use this directly, as it is quite generic, the answer was actually pretty good as a starting point.

I was now hooked, and I had to keep going on now.

What test cases should I automate for my product?

Example: What test cases should I automate for Amazon India?

Like the test strategy, these are not to the level of details I am looking for. But it’s a great start.

That said, at the top of the test automation pyramid, I do not want to automate test cases, but I want to automate test scenarios. So, I asked ChatGPT about the important scenarios I should automate.

What test scenarios should I automate for my product?

Though not the specific test scenarios I was expecting, there is a clear difference between the identified test cases and the test scenarios.

Code generation: the first auto-generated test

I asked ChatGPT to give me the Selenium-Java automation code to automate the “search and add a One Plus phone to cart scenario” for Amazon India website.

ChatGPT not only corrected my question, but also gave me the exact code to implement the above scenario.

I updated the question to generate the test using Applitools Visual AI, and voila, here was the solution:

Is this code usable?

Not really!

If you run the test, there is a good chance it would fail. The reason being, the generated code assumes the page is “rendered” as soon as the actions are done. To make this code usable and consistent in execution, we need to update it with realistic and intelligent waits at relevant places to cater to the rendering and loading time required by the browser, and of-course, your environment.

NOTE: Intelligent waits are important as opposed to a hard wait to optimise the execution time. Also, being realistic in your wait durations is very important. You do not want to wait for a particular state of the application for more than what is expected to happen in production.

Also, typically in our test frameworks, we have various abstraction layers and utilities to manage different functionalities and responsibilities. But we can use snippets of this generated code and easily plug it in the right places in our frameworks. So it is still very valuable.

I also tried the same for some internal sites – i.e. not available to the general public. The test was still created, but not sure about the validity of those tests.

Test data generation

Lastly, I asked specific questions around test data – an aspect unfortunately ignored or forgotten until the last minute by most teams.

Example: What test data do I need to automate test scenarios for my product?

Again, I was blown away by the detailed test data characteristics that were given as a response. It was specific about the relevant data required for product, user, payment, and order to automate the important test scenarios that were highlighted in the earlier response. 

I tried to trick ChatGPT to “search and add a product to cart” – but I didn’t specify what product.

It caught my bluff and gave very clear instructions that I should provide valid test data.

Try it yourself: Can ChatGPT accurately translate your tests from one framework to another? Try it out and let us know how it went @Applitools.

Other examples

Generating content

Instead of thinking and typing this blog post, I could have used ChatGPT to generate the post for me as well.

I didn’t even have time to step away to get a cup of coffee while this was being generated!

Planning a vacation

Another interesting use case of ChatGPT – getting help to plan a vacation!

NOTE: Be sure to work with your organization’s security team and @OpenAI before implementing ChatGPT into the organization’s official processes.

What’s next? What does it mean for using AI in testing?

In my examples, the IDEs are very powerful and make it very easy for programmers and automation engineers to write code in a better and more efficient way.

36.8% of our audience participants answered that they were worried about AI replacing their jobs. Others see AI as a tool to improve their skills and efficiency in their job.

Tools and technologies like ChatGPT will offer more assistance to such roles. Individuals can focus on logic and use cases. I have doubts on the applicability of these tools to provide answers based on contextual information. However, given very focused and precise questions, tools can provide information to help implement the same in an efficient and optimal fashion.

I am hooked!

To learn more, watch on-demand recording of Unlocking the Power of ChatGPT and AI in Testing: A Real-World Look. Be looking for a follow-on event in early April where I will go deeper into specific use cases and how ChatGPT and its use in testing are evolving. Sign up for an email notification when registration opens.

Where to start from?

If you are a new manager or willing to be promoted to a managerial role, you could start from the following options:

•  Research the role fully and understand what your deliverables, career progression, and path look like.
• Look for some management training internally/externally to see if this is the right “future” role for you.
•  Speak to existing managers and see what their thoughts are, where did they start from, and if they have any tips or if you could shadow them with a task or two.
• Attend talks/listen to podcasts from existing managers and see if you like the sound of the role.

The above steps could help you land on your dream job.

“There are over 1,583,412 managers currently employed in the United States. 49.0% of all managers are women, while 51.0% are men. The average age of an employed manager is 44 years old.”

Zippia

Furthermore, something to be more involved with could be understanding and establishing a leadership philosophy within your practise and seeing what the future of this practise is. Focusing on the day to day of management and leadership in your current role/remit is also a starting point. Be clear about your communication and top priorities as this will also help you engage better as a manager. Setting common values and common standards will also add to your role and the most important thing, IT IS OK TO BE SCARED/VULNERABLE or even go through Imposter Syndrome. Every manager I have come across is not perfect. There is no such thing as a perfect manager. However, a manager must not only focus on technical deliveries but also be a people’s person first and foremost. It’s all about the team and with the team you make the dream work successfully and confidently.

Traits to adopt

Be a good coach because this role is not just about excelling and delivering, but also coaching those who would work with you or aspire to have a similar role in the near future. Coaching is so beautiful. It’s amazing how we can ask open-ended questions and untangle individuals to reach their answers in a quicker manner.

Express interest/concern for team members’ success and personal well-being, because it’s about your team. It’s vital to celebrate your team as well as focus on their well-being. We work day in and day out, sometimes we forget to take a break. Sometimes I have my team members come up to me asking me if there is any other piece of work, I just tell them “use this time for self-learning, relax, and enjoy this free time”. Something as simple as this brings a smile on their faces. The world is changing, and I feel we should move with it. Never fall into the trap of micro-management. Trust, value, and celebrate your team.

Aim for a productive and results-orientated managerial way of working. This will help you showcase your teams, your work, and your ways of working. Serve as an example. Sometimes we delve into deliverables too much and have no time to showcase any of the results we achieve. To get good results, we also need to work on a process/strategy to get us what we are aiming to reap.

Empower the team and do not micromanage. I think this is self-explanatory. If you empower your team, it’s very common to see the results you all want to see whether that’s winning more projects to test, great testing feedback, or your WoW being commendable. Micromanaging doesn’t get us anywhere but at the doorstep of HR. Whereas being sociable as a manager gives your organization a friendly face, making employees feel more secure at work. Happy members of staff work harder, and are less likely to produce substandard results because they are just trying to get through the day.

Be a good communicator and listen and share the right information. When you come to know something vital for your team, make sure to communicate and involve them. Allow them to also express their thoughts. Differences in every team exist, but you call that a great team. Not everyone has to have to think in the same manner, which might help trigger an important issue no one else could think of. Also, listening to everyone, making time for this task is another super important trait. It’s not always about doing but also listening.

Just like you became a manager, your team would also have career aspirations, therefore help the team with their objectives and career development with the right training paths, learnings, and experiences.

As I mentioned earlier there is no perfect/right type of manager. It’s all about your learnings, skillset, and expertise to excel within this role. You can obviously adapt the above and excel more.

The importance of being a People Manager

“This position allows you to ‘sense’ every member of the team as well as the shape of the team itself. Sensing all members refers to the first statement above, getting to know your team members, listening to them, understanding who they are, how they feel, how they fulfill their part of the work within the team.”

Jean-Philippe Grenet

Effective people management can improve morale and employee engagement, reduce turnover, and enhance communication throughout teams. Furthermore, being there for your team means you can optimise productivity, engage employees, and develop a culture of innovation.

A good leader gets to know their team and makes time for them, whether that’s for some feedback, general chat, or even a serious topic. When an employee is going through something in their personal or professional life, extending empathy and compassion is a very ensuring act from a manager. It helps bring focus and value within a team.

Remember also to give and receive feedback. As a leader, your team is working with you to support the mindset and quality delivery you and the entire team have planned and had an input in. Feedback is expected and is an excellent way to revert back to members on how they are doing. Sometimes feedback can be a little difficult to deliver if it’s not all positive, but which team is 24/7 and every year working as a star team? Perhaps, choose a different place other than the office to speak to a team member and provide them with some guidelines, support, and time. On the other hand remember to take feedback too, on how your team feels you are working with them. It’s a two-way process. 

Therefore, being there for your team would lead to effective people management and can help boost employee retention, employee engagement, and organisational effectiveness.

Conclusion

A manager in my eyes is one who not only focuses and is responsible about test best practices, but leads with a technical mindset too. A good manager’s door will always be open regardless of how busy he/she is. Also being sociable as a manager gives your org a friendly face and makes employees feel more secure. Do not forget to empathize and be a compassionate lead as this helps the team’s well-being too and in turn, there is respect and transparency. Finally, aim for a collaborative and holistic working approach as that would not only help your team grow but also serve as a great example.

Be sure to follow my social channels:

• LinkedIn
• Twitter
• Website

Learn how to future-proof your test automation pipeline with Cypress and Applitools by adding tests that run from GitHub Actions. In this article, we’ll share how to ensure your test automation pipeline can scale while staying reliable and easy to maintain.

Automating different types of tests

To illustrate our different types of test automation, we’ll be using the example project Cypress Heroes. In this full-stack TypeScript app, users can take the following actions:

• Log in with an email and password
• Like heroes, which increments the hero’s number of fans
• Hire heroes, which increments the hero’s number of saves
• Manage hero profile information like name, superpowers, and price

ICYMI: Watch the on-demand recording of Future-Proofing Your Automation Pipeline to see Ely Lucas from Cypress demo the example project.

End-to-end testing

Cypress is traditionally known for end-to-end testing. You automate user interactions for specific scenarios from start to finish in the browser, and then run functional assertions to check the state of elements at each step. End-to-end tests are hidden in an actual web server and hit the site just like a user would.

Measurable stats for code coverage of your end-to-end testing can act as a health metric for your website or app. Adding coverage reports to your automation pipeline as commits can help ensure you’re testing all parts of your code.

Component testing

If you’re using a component-based framework like React or Angular or a design system like Storybook, you can also do component testing to test UI components. In this example, we have a button component with a few tests that pass, the hero card test, and a test for the login form. These components are being mounted in isolation outside of your typical web server.

Think of component tests as “UI unit” tests. While they don’t give end-to-end coverage, they’re quick and easy to run.

API testing

For your back end, you’ll need to automate API tests. The example project is using a community-built plugin called cypress-plugin-api. This plugin provides an interface inside the Cypress app to test APIs. It’s really cool and it’s super fun, and it allows you to write tests that you would have to do manually in a tool like Postman.

Fun fact: Cypress Ambassador Filip Hric developed the cypress-plugin-api. Check out Filip’s Test Automation University courses.

The API tests in our example are in the separate server project. We can use the command npx cypress open, and then we can run those tests in Chrome. We can see all of our results that we’re getting the response of the status codes. We can view a post request, the headers that were sent, the headers that were returned, and other stuff that you normally get from a tool like Postman.

And it’s just baked into the app, which is really nice. Cypress is basically a web app that tests a web app. And so, you could extend Cypress as an app with things like this to help you do your testing and to have it all seamlessly integrated.

Running a pipeline with GitHub Actions

The example project uses GitHub Actions to set up the test automation pipeline. When working on smaller projects, it’s easy to have CI interactions baked into your repository, all in one place.

Configuring your GitHub Action

With GitHub Actions, you declare everything you need in a YAML file in the .github/workflows folder. Your actions become part of your repository and are covered by version control. If you make any changes, you can review them easily with a simple line-by-line diff. GitHub Actions make it easy to automate processes alongside other interactions you make with your repository. For example, if you open a pull request, you can have it automatically kick off your tests and do linting. You can even perform static code analysis before merging changes.

Some environment variables are set at the top of the YAML file. The API URL is what the client app uses to communicate to the API. The example app is hooked up to send test results to the Cypress Cloud. Those results can then be used for analytics, diagnostics, reporting, and optimizing our test workflows. The Cypress cloud also requires a GitHub token, so it can do things like correctly identify what pull request is being merged.

For those new to GitHub Actions: You can define environment variables per step in a job, but declaring them at the top helps you update them painlessly.

Running each of our tests

To keep things simple, there is only one job right now in this GitHub Action. First, it checks out the code straight from GitHub. Next, it builds the project using the Cypress GitHub Action. The Cypress GitHub Action does a few things for you like building your application or npm installing or yarn installing the dependencies.

Building first means that subsequent jobs don’t have to build the app again. We’ve set run test to false, which is a parameter to the Cypress GitHub Action, because we don’t want to run the tests here. We’ll be running the tests separately below.

We have our component tests in our GitHub Action. We tell it to install false, since we installed it up above. And then we run our custom test command, which opens Cypress in run mode and initiates component testing. This record tells the GitHub Action to send the results to Cypress Cloud.

And then we have to start the client and server. For both end-to-end tests and API tests, the application must be up and running component tests. For the end-to-end test and API tests, the example app is hitting live servers.

This run command will start both the React app and the Node server, and then it will run the end-to-end test. We’re telling it again to not install the dependency, since it was already installed. Then, we’re running the command to start the end-to-end testing. The wait command will wait to make sure that both the client URL and the API URL are both up and running before it will start the test. If the test starts before both URLs get up and running, you’ll have some tests fail.

Another thing that the Cypress GitHub Action does is that you have the option to wait for these services to be live before the testing starts. By default, the npm run test commands are going to use the Chromium browser built into Electron. If you want to test on other browsers, you must make sure those browsers are installed on the runner. Cypress provides Docker images that you can add to your configuration to download the different browsers. However, downloading additional browsers increases the file size and makes the runs take longer.

Make sure that the Cypress binary itself is downloaded and installed. It’s going to run headlessly. This is because the command set up in these scripts is run mode, which is headless, whereas open mode is with the UI.

And then it will run the API test, which is very similar to end-to-end tests, except that since we’re not hitting the actual client app – only hitting the API app – we’re only waiting to make sure that the API URL is up and running.

If you write test cases per the local database for end-to-end testing before pushing to GitHub Actions, someone else running those test cases on their system could potentially fail. In whatever kind of test automation you develop, you’ll need to handle test data properly to avoid collisions. There are many different strategies you can follow. For more information on this and solving sample data dependency, watch my talk Managing the Test Data Nightmare.

How long do the test suites take?

When running your tests with Cypress and GitHub Actions, the results are uploaded to Cypress Cloud. You can go into Cypress Cloud and actually watch replays of all these tests that happened. The entire pipeline run in the example was 3 minutes and 50 seconds for all three test suites.

The individual test suites we ran took the following times:

• Component tests: 49 seconds
• End-to-end tests: 1 minute and 18 seconds
• API tests: 13 seconds

Improving test coverage with visual assertions

Since all the Cypress tests are run inside of the browser window, you can visually see them and inspect to make sure that they’re looking correctly. But this type of review is a manual step. If someone accidentally makes a change to the stylesheet, the site could no longer be running properly, but if we run the tests, they’ll pass.

We can use Applitools Eyes to fix this issue.

Visual testing is meant to automate the things that traditional automation is not so good at. For example, as long as particular IDs on your page are in the DOM somewhere, your traditional automation scripts with something like Cypress are still going to find and interact with the elements. Applitools Eyes uses visual AI to look at an app and be able to detect these kinds of visual differences that traditional assertions struggle to capture. Let’s add some visual snapshots to these end-to-end tests.

Adding Applitools to your project

First, you’ll need an Applitools account. You can register a free Applitools account with your GitHub username or your email, and you’ll be good to go. You’ll need your Applitools API key for when we run tests with Applitools.

Next, we’ll need to install the Applitools SDK using npm install @applitools/eyes-cypress.

It can be a dev dependency or it can be a regular dependency – whichever you prefer. In the example project, we use a dev dependency. In the example project, we’re using the Applitools Eyes SDK for Cypress, we have Applitools SDKs for basically every tool framework you got.
Next, we’ll need to create an Applitools configuration file. Where in Cypress projects, you have your cypress.config.js file, basically we want one that’s called applitools.config.js.

Configuring your Applitools runner

In the Applitools config file, we will specify the configuration for running visual tests. There’s a separation between declaring configuration and actually adding test steps.
One of the settings we want is called batchName, and we’re going to set that to “cy heroes visual tests” to reflect the name of our demo app. The batch name will appear in the Eyes Test Manager (or the Applitools “dashboard”) after we run our visual tests.

Next, we’ll set the browsers. This will be a list, with each item being an entry that specifies a browser configuration, including name, width, and height.

Typically, since Cypress runs inside of an Electron app, it can be challenging to test mobile browsers. However, the Applitools Ultrafast Grid enables us to render our visual snapshots on mobile devices. The settings for mobile devices are going to be a bit different than those for browsers. Instead of having a name, we’re going to have a device name.

Our applitools.config.js file is complete. When we run our tests – either locally or in the GitHub Action – Applitools will render the snapshots it captures on these four browser configurations in the Ultrafast Grid and report results using the batch name. Furthermore, the local platform doesn’t matter. Even if you run this test on Windows, the Ultrafast Grid can still render snapshots on Safari and mobile emulators. A snapshot is just going to be a capture of that full page. Applitools will do the re-rendering with the appropriate size, the appropriate browser configuration, and all that will happen in the cloud. Essentially you can do multi-browser and multi-platform testing with simple declarations.

Now that we have completed the configuration, let’s update the tests to capture visual snapshots, starting with the homepage.

Setting up our test suites

You need to make sure that your tests aren’t interfering with other tests. In these tests, we’re going through and modifying some of the heroes that are in the application. The state of the application changes per test, so to get around that, we’re creating a new hero just for working with our tests and deleting the hero after the tests.

In the example, we’re using Cypress tasks, which is code that actually runs on the Node process part of Cypress. It’s directly communicating with our database to add the hero, delete the hero, and all the other types of setup tasks that we want to do before we actually run our test.

So it’s going to happen for each of the tests, and then we’re visiting the homepage and getting access to the hero.

We get our new hero and then we call cy.deleteHero, which is going to call the database to delete the hero. From the describe block at the start of every test, we get our hero. And then, finally, we have the hero card by its name, and we find the button that has the right selectors, so we can actually select it and click the button.

This test is making sure that you’re logged in before you can like this hero. We’re making sure that the modal popped up, clicking the okay on the modal, and then making sure that modal disappears and does not exist anymore.

Down below we have another suite for when a normal user is logged in. And so we’re using a custom Cypress command to log in with this username and password. You can define these custom commands that are like making your own function, encapsulating a little bit of logic so that it could be reusable.

So what we’re doing to test the login is going to the homepage, running the login process, and verifying the login was actually successful. The cy.session is caching a session for us to restore the session later from cookies. This helps speed up your test so you’re not having to go through the whole flow of actually logging in again.

We have another suite here for when an admin user is logged in, because an admin user can edit users and delete heroes.

 In the example, negative login tests – where you use the wrong username and/or password – are under the component tests.

In the login form component test, when an email and password is invalid, an error should show. The example uses cy.intercept to mock the API request that goes to the cert, which goes to the off endpoint and returns a status code 401, which represents an invalid login.

You can either write a component test or an end-to-end test. In this case, a component test makes it easier to set up the mock data.

Adding a call to Applitools Eyes

With the test suites set up, we’re ready to add some visual snapshots here. We need to call an Eyes session using the Applitools Eyes SDK. The idea is that we open our eyes, and we can take visual snapshots. And then at the end of the test, we will close our eyes to say that we’ve captured all the snapshots for that session or for that test. And at that point, Applitools Eyes will upload the snapshots that are captured to the Applitools Eyes server, do all of the re-rendering of the things of those four browsers in the Ultrafast Grid. Then we can log into the Applitools dashboard and we can see exactly what happened with our testing.
To get the autocomplete for Eyes commands, we need to set up the Applitools Eyes stuff with the Cypress project. We already did npm install on the package, so we’ll need to run npx eyes-setup.

We’ll want to use the command cy.eyesOpen in the homepage describe block under the beforeEach method. We want to pass an app name and test name for logging and reporting purposes. You might also put their Cypress eyes open code in the beforeEach of the test cases, so the call doesn’t need to be duplicated.

Then, in the afterEach block, you’ll call cy.eyesClose.

In this test, you must log in, make sure that the modal pops up, log in, and then click okay in the modal and make sure the modal disappears, so we’ll need a snapshot when the modal is up and one when the modal goes away. In this case, we’ll capture the whole window.

If we didn’t want to capture everything, we could actually capture a region, like a div or even an individual element. On a small scale, using the region option does not make a measurable difference in execution speed, but it gives you a way to tune the type of snapshot we want.

For capturing the next step, we can basically copy the whole call there and paste it, changing the tag to homepage with the modal dismissed.

These snapshots are very straightforward to write, and something that we could consider is that some of those other assertions you might arguably be able to remove. The visual snapshot is going to capture everything on that window, so if it’s there and visible, we’re going to capture it and track it over time.

You would still need to keep all of your interactions, but you can remove most of your assertions checking visible elements. However, there are certainly things where if you want to check a very specific numeric value, you still want to keep those assertions.

Running the updated tests

All we need to do to run this test is make sure that we have our Applitools API key from our account saved as an environment variable of the Cypress application.

Note: If you happen to steal someone’s API key, it doesn’t really help you. It just means they’ll see your results, and you won’t. API keys should be kept secret and safe.

Using the Applitools Eyes dashboard

So to see the visual testing results, we will need to view them in the Applitools Eyes dashboard.

You can view your test results in a grid to see the UI quickly, or you can view your results in a list to see your configurations quickly.

On the left, you’ve got the batch name that was set. Then on the main part of the body, you’ll see there are actually four tests. We only wrote one test, but each test is run once per browser configuration we specified, providing cross-browser and cross-platform testing without additional steps.

If we open up the snapshots, you can see the two snapshots that we captured. These results are new, because this is the first time we’ve run the test.

We’ve established the snapshot as a baseline image, meaning anything in the future will be checked against that.

That’s where that visual aspect of the testing comes in. Your Cypress results will essentially tell you if it was bare bones basic functional, and then Eyes will tell you what it actually looked like. You get richer results together.

Resolving test results in the dashboard

Let’s see what this looks like if we make that visual change.

In the main file, we’ve updated the stylesheet and run the test again. There is no need to do anything in the Applitools Eyes dashboard before re-running the test.

The new test batch is in an unresolved state because Eyes detected a visual difference. In theory, a visual difference could be good or bad. You could be making an intentional change. Visual AI is basically a change detector that makes it obvious to you, the human, to decide what is good or bad. Then anytime Applitools Eyes sees the same kind of passing or failing behavior in the future, it’ll remember.

It’s important to note that the unresolved test results won’t stop your test automation job or your automation flow. Test automation would complete normally. You as the human tester would review visual test results in the Eyes Test Manager (the “dashboard”) afterwards. The pipeline would not wait for you to manually mark visual test results.

Let’s open up one of those snapshots so we can see it full screen.

In the upper left, below the View menu in the ribbon, there’s a dropdown to show both so that you can see the baseline and test side by side.

In the example, we had removed the stylesheet, so we can see very clearly that it’s very different. It’s not always this obvious. In this case, pretty much the whole screen is different. But if it were like a single button that was missing or something shunted a little bit, it would show that a specific area was different. That’s the power of the visual AI check.

Whenever Applitools detects a visual change, you can mark it as “passing” with a thumbs-up. Then that snapshot automatically becomes the new baseline against which future checkpoints are compared. Applitools will go to the background and track similar images. And it will automatically update those appropriately as well.

Note: If you ever want to “reset” snapshots, you can also delete the baselines and run your tests “fresh” as if for the first time. The snapshots they capture will automatically become new baseline images.

Once we’ve resolved all test results, we’ll need to save. And now if we were to rerun our test again, Applitools Eyes would see the new snapshots and pass tests as appropriate. If you have dynamic content or test data, you add region annotations, which will ignore anything in the region box.
It is possible to compare your production and staging environments. You can use our GitHub Integration to manage different branches or versions of your application. We also support different baselines for A/B testing.

Closing thoughts

That’s basically how you would do visual testing with Applitools and Cypress. There are two big points to remember if you want to add visual testing to your own test suites:

• To get these tests running in your pipeline, the only change you’d have to make is to inject the Applitools API key in those environment variables.
• We didn’t really add a fourth suite of tests. Visual testing is more of a technique or an aspect of testing, not necessarily its own category of tests. All you have to do is work in the SDK, capture some snapshots, and you’re good to roll.

We hope this guide has helped you to build out your test automation pipeline to be more reliable and scalable. If you liked the guide, check out our Applitools tutorials for other guides on building your test automation pipeline. Watch the on-demand recording of Future-Proofing Your Automation Pipeline to see the full walkthrough. To keep up-to-date with test automation, you can peruse our latest courses taught by industry-leading testing experts on Test Automation University. Happy testing!

Let the Engineers Speak! Part 5: Audience Q&A

Articles — 01.11.2023

Let the Engineers Speak! Part 4: Changing Frameworks

Articles — 01.05.2023

Let the Engineers Speak! Part 3: Favorite Test Integrations

Articles — 12.28.2022

The AI promise

AI as a technology is not new. There are huge advancements made in the recent past in this field. 

Currently it seems like AI technology is mostly about using ML (machine learning) algorithms to train models with large volumes of data and use these trained computational models to make predictions or generate some outcomes.

From a testing and test automation point-of-view, the question in my mind still remains: Will AI be able to automatically generate and update test cases? Find contextual defects in the product? Inspect code and the test coverage to prevent defects getting into production?

These are the questions I have targeted to answer in this post.

The hype

Gartner publishes reports related to the Hype Cycle of AI. You can read the detailed reports from 2019 and 2021.

The image below is from the Hype Cycle for AI, 2021 report.

Based on the report from 2021, we seem to be some distance away from the technology being able to satisfactorily answer the questions in my mind. 

But there are areas where we seem to be getting out of what the Gartner report calls “Trough of Disillusionment” and moving towards the “Slope of Enlightenment”.

Based on my research in the area, I agree with this.

Let’s explore this further.

The promising future

Recently I came across a lot of interesting buzz created by ChatGPT, and I had to try it out. 
Given I am a curious tester by heart, I signed up for it on https://chat.openai.com/ and tried to see the kind of responses generated by ChatGPT for use cases related to test strategy and automation.

The role and value of AI in testing

I took ChatGPT for a spin to see how it could help in creating test strategy and generate code to automate some tests. In fact, I also tried creating content using ChatGPT. Look out for my next blog for the details of the experiment.

I am amazed to see how far ahead we have come in making the use of AI technology accessible to the end-users.

The answers to a lot of the non-coding questions, though generic, were pretty spot on in the responses. Like the code generated by record-and-playback tools cannot be used directly and needs to be tuned and optimized for regular usage, the answers provided by ChatGPT can be a great starting point for anyone. You would get a high-level structure, with major areas identified, which you can then detail out based on context, that only you would be aware of.

But, I often wonder what is the role of AI in Testing? Is it just a buzzword gone viral on social media?

I did some research in the area, but most of the references I found were blog posts and articles about “how AI can help make testing better”.

Here is a summary from my research on trends and hopes from AI & Testing.

Test script generation

This aspect was pretty amazing to me. The code generated was usable, though in isolation. Typically in our automation framework, we have a particular architecture we conform to while implementing the test scripts. The code generated for specific use cases will need to be optimized and refactored to fit in the framework architecture.

Test script generation using Applitools Visual AI

Applitools Visual AI takes a very interesting approach at this. Let me explain this in more detail.

There are 2 main aspects in a test script implementation.

1. Implementing the actions/navigations/interactions with the product-under-test
2. Implementing various assertions based on the above interactions to validate the functionality is working as expected

We typically only write the obvious and important assertions in the test. This itself can make the code quite complex and also affects the test execution speed. 

The less important assertions are typically ignored – for lack of time, or also because they may not be directly related to the intent of the test.

For the assertions you have not added in the test script, you either need to implement different tests for validate those, or worse yet, they are ignored.

There is another category of assertions which are important and you need to implement in your test, but these are very difficult or impossible to implement in your test scripts (based on your automation toolset). Example – validating the colors, fonts, page layouts, overlapping content, images, etc. is very difficult (if at all possible) and complex to implement. For these types of validations, you usually end up relying on a human testing these specific details manually. Given the error-prone nature of manual testing, a lot of these validations are often (unintentionally) missed or ignored and the incorrect behavior of your application gets released to Production.

This is where Applitools Visual AI comes to the rescue.

Instead of implementing a lot of assertions, with Applitools Visual AI integrated in your UI automation framework, it can take care of all your functional and visual (UI and UX) assertions with a single line of code. With this single line of code, you are able to check the full screen automatically, thus exponentially increasing your test coverage. 

Let us see an example of what this means.

Below is a simple “invalid-login” test with assertions.

If you use Applitools Visual AI, the same test changes as below:

NOTE: In the above example with Applitools Visual AI, the lines eyes.open and eyes.closeAsync are typically called from your before and after test method hooks in your automation framework.

You can see the vast reduction in code required for the same “invalid-login” test as compared between the regular assertion-based code we are used to writing, versus using Applitools Visual AI. Hence, the time to implement the test has improved significantly.

The true value is seen when the test runs between different builds of your product which has changes in functionality.

The standard assertion-based test will fail at the first error it encountered. Whereas the test with Applitools is able to highlight all the differences between the 2 builds, which include:

• Broken functionality – bugs
• Missing images
• Overlapping content
• Changed/new features in the new UI

All the above is done without having to rely on locators, which could have changed as well, and caused your test to fail for a different reason. It is all possible because of the 99.9999% accurate AI algorithms provided by Applitools for visual comparison. 

Thus, your test coverage has increased, and it is impossible for bugs to escape your attention.

The Applitools AI algorithms can be used in any combination as appropriate to the context of your application to get the maximum validation as possible.

Based on the data from the survey done on “The impact of Visual AI on Test Automation”, we can see below the advantages of using Applitools Visual AI.

Increasing test coverage

AI technology is getting pretty good in suggesting test scenarios and test cases. 

There are many products that are already using AI technology under the hoods to provide valuable features to the users.

Increasing test coverage using Applitools Ultrafast Test Cloud

A great example of an AI-based tool is the Applitools Test Cloud which allows you to scale the test execution seamlessly without having to run the tests on other browsers and devices.

In the image below, you can see how easy it is to scale your web-based test execution across the devices and browsers of your choice as part of the same UI test execution using the Applitools Ultrafast Grid.

Similarly you can use the Applitools Ultrafast Native Mobile Grid for scaling the test execution for your iOS and Android native applications as shown below:.

Example of using Applitools Ultrafast Native Mobile Grid for Android apps:

Example of using Applitools Ultrafast Native Mobile Grid for iOS apps:

Test data generation

This is an area that is still to get better. My experiments showed that while actual test data was not generated, it indicated all the important areas we need to think about from a test data perspective. This is a great starting point for any team, while ensuring no obvious area is missed out.

Debugging

We use code quality checks in our code-base to ensure the code is of high quality. While the code quality tool itself usually provides suggestions on how to fix the flagged issues, there are times when it may be tricky to fix the problem. This is an area where I got lot of help in fixing the problem.

Example: In one of my projects, I was getting a sonar error related to “XML transformers should be secured”.

​​I asked ChatGPT this question:

The solution to the problem was spot-on and I was immediately able to resolve my sonar error.

Code optimization and fixing

Given a block of code, I was able to use ChatGPT to suggest a better way to implement that code, without me providing any context of my tech stack. This was mind-blowing!

Here is an example of this:

There were 5 very valuable suggestions provided by ChatGPT. All of these suggestions are very actionable, hence very valuable!

Analysis and prediction

This is a part I feel tools are currently still limited. There is an aspect of product context, learning, and using the product and team-specific data to come up with very contextual analysis and eventually get to the predictive and recommendations stage.

What other AI tools exist?

While most of the examples are based on ChatGPT, there are other tools also in the works from a research perspective, and also application perspective.

Here are some interesting examples and resources for you to investigate more in the fascinating work that is going on.

• Galactica: Meta’s Answer to GPT-3 for Science
• Sparrow: Improving alignment of dialogue agents via targeted human judgements
• GODEL (Grounded Open Dialogue Language Model): Large-Scale Pre-training for Goal-Directed Dialog
• DialoGPT: Toward Human-Quality Conversational Response Generation via Large-Scale Pretraining
• BERT (Bidirectional Encoder Representations from Transformers): Pre-training of Deep Bidirectional Transformers for Language Understanding
• Pathways Language Model (PaLM): Scaling to 540 Billion Parameters for Breakthrough Performance
• Pathways: A next-generation AI architecture
• Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
• Scaling Language Models: Methods, Analysis & Insights from Training Gopher
• LaMDA: Towards Safe, Grounded, and High-Quality Dialog Models for Everything
• More Efficient In-Context Learning with GLaM
• BlenderBOT: Multi-Modal Open-Domain Dialogue
• Amazon’s Alexa Teacher Models
• Whisper: That approaches human level robustness and accuracy on English speech recognition
• DALL-E: Generate images

While the above list is not exhaustive, it is more focused in the area of research and development.

There are tools that leverage AI and ML already providing value to users. Some of these tools are:

• Applitools Visual AI
• QuillBot’s AI-powered paraphrasing tool to enhance your writing
• Out of office email generator
• Viral Post Generator

Testing AI systems

The tester in me is also thinking about another aspect. How would I test AI systems? How can I contribute to building and helping the technology and tools move to the Slope of Enlightenment in Gartner’s Hype Cycle? I am very eager to learn and grow in this exciting time!

Challenges of testing AI systems

Given I do not have much experience in this (yet), there are various challenges that come to my  mind to test AI systems. But by now I am getting lazy to type, so I asked ChatGPT to list out the challenges of testing AI systems – and it did a better job at articulating my thoughts … almost felt like it read my mind.

This answer seems quite to the point, but also generic. In this case, I also do not have any specific points I can think of, given my lack of knowledge and experience in this field. So I am not disappointed.

Why generic? Because I have certain thoughts, ideas, and context in my mind. I am trying to build a story around that. ChatGPT does not have an insight into “my” thoughts and ideas. It did a great job giving responses based on the data it was trained on, keeping the complexity, adaptability, transparency, bias, and diversity in consideration.

For fun, I asked ChatGPT: “How to test ChatGPT?”

The answer was not bad at all. In fact, it gave a specific example of how to test it. I am going to give it a try. What about you?

What’s next? What does it mean for using AI in testing?

There is a lot of amazing work happening in the field. I am very happy to see this happen, and I look forward to finding ways to contribute to its evolution. The great thing is that this is not about technology, but how technology can solve some difficult problems for the users.

From a testing perspective, there is some value we can start leveraging from the current set of AI tools. But there is also a lot more to be done and achieved here!

Sign up for my webinar Unlocking the Power of ChatGPT and AI in Testing: A Real-World Look to learn more about the uses of AI in testing.

Learn more about visual AI with Applitools Eyes or reach out to our sales team.

There are many metrics that drive the efficiency of an engineering team. These are easier to meet when your team is small but after the team crosses 50 engineers, it is reasonably hard to manage engineering productivity. Most engineering managers spend all their time ensuring that the team does not have bottlenecks. The north star for teams is usually some well defined metrics at this stage. We interviewed a group of 20 engineering managers from leading companies in Australia and India to find the ones that are really important to their success. These are the ones we found are most important to them.

Cycle Time

Cycle time is a universal engineering metric that determines how effective a team is. A group must spend a certain amount of time on a feature from start to finish. Usually this includes planning, development, and testing. The metrics measure how quickly the development team can deliver the feature but it may not necessarily be deployed to production. 

Faster cycle time is a goal for every development team. The ability to monitor cycle times allows engineering managers to identify potential bottlenecks in the delivery process. A lot of compromises are sometimes made to meet higher cycle time as agility is very important to every business today.

Deploy Frequency

You can determine how often your team can release code into production by calculating the deployment frequency. Note that cycle time does not include deployment time. Teams working on development aim to distribute smaller pieces of code more frequently and in smaller batches.

It allows deployments to be more manageable to test and release. It also improves your overall efficiency.

Rework Ratio

This appears to be the most important metric for many teams and happens to also be a big area of concern. The rework ratio indicates the amount of code that must be changed after the team delivers it to production. The rework can be a bug or feature enhancement. If you have a high rework percentage, it can reduce your overall efficiency. 

Meeting high deployment frequency and cycle time can create an impact on the amount of testing done to push to production. This can lead to higher rework ratio as issues get raised by users later in the cycle. Any bug raised later leads to lost time in fixing old code which reduces the overall efficiency of the team. 

An insufficient level of communication or a flawed review process could lead to quality issues in the future.

Context Switching

As a result of various obstacles, team members must switch between issues in context. When the team switches context frequently, they are not working efficiently. To maintain focus, the appropriate adjustments should be made in this case. A huge reason for context switching is the process around fixing bugs in the development process. Sometimes the development process makes development teams use tools that make it hard to remain in the context of the development process. Most of the times, it is the testing cycle that leads to context switching due to the lack of adequate integrations across the development lifecycle. 

As you may have observed, there is one common thing that we found becomes an obstacle in achieving better engineering productivity metrics. The desire to drop cycle time and deploy frequently is usually done at a compromise of testing coverage. Eventually, it leads to more rework ratio and more context switching for the teams. Most teams that scaled their engineering process start by paying acute attention to the testing process. The idea is to automate what can be automated with tools that can allow developers to move faster.

Testing fast and at scale is the key to engineering efficiency. Spotify coined the term for this called “Quality at Speed”. To maintain Quality at Speed, a Quality Engineering counterforce is required. At Applitools, our customers have helped in achieving quality at ultrafast speed. Visual AI from Applitools provides you with the ability to extend human eyes on the testing process without having to increase the QA/Dev ratio in the team.  Some quotes from engineering managers that have used Applitools for building products include:

“Any engineering team can reduce the manual testing resource and time by at least 70%. It also avoids overloading of SDETs.”

Engineering Manager at Dunzo (India)

“At Pushpay, our success stems from a technology-forward culture which drives our behavior, how we solve problems, and what tools we use to solve them. Since partnering with Applitools over 5 years ago, we have been able to improve quality, gain productivity and thus save time and money. We could not be more pleased with the efficiency boost our team has experienced since adopting Applitools and more recently, the Ultrafast Grid.”

Engineering Manager at Pushpay (New Zealand)

Using visual AI to address these factors

If the above is something you wish to improve within your team, then you will be surprised that it takes just a few days to get to this degree of speed at quality. 

The picture below shows how Applitools integrates with your application.

I will not get into how to install Applitools, as that is fairly well described in the tutorials. This also includes how you can integrate Applitools within your CI/CD pipeline. In the remainder of this article, I will like to tell you about some great examples of improving these metrics using Applitools. 

Most efficient teams would start a day with the below dashboard. This gives you a comprehensive view of all your tests that have been executed across your entire coverage list. Having high coverage is made easy by using an ultrafast grid that reduces your rework ratio later for devices or browsers that may not have been included before. At last a number of re-work happens due to poor coverage of testing in the first phase of development. Of course there is an element of scope creep that leads to the re-work as well which can be easily avoided by involving cross-functional teams in the development process. Applitools provide a visual abstraction of your application that can be accessed by everyone on a GUI. This drastically brings down the areas of scope creep unless the requirements have totally changed by business.

When you are doing testing for high coverage it becomes important that you are not getting slowed down by the process of reviewing the bugs. A big reason why deployment frequency gets reduced is because of the time it takes to review and fix the issues. This is exactly where Visual AI plays a big role.

Visual AI also lets you troubleshoot the defects really quickly.

Finally, developers and testers can use the same platform which integrates seamlessly with Jira or their preferred communication channel for faster feedback. Email and Slack notifications help the team get the feedback fast without any context switching.

To conclude, the engineering manager needs to explore more deeply how engineering processes are structured as the engineering team grows. Businesses are demanding faster releases of standard quality products, and Visual AI is an effective method of improving both efficiency and coverage of testing.

Learn about more about visual AI with Applitools Eyes.

This article is based on our recent webinar, How to Enhance UI/UX Testing by Leveraging AI, led by Chris Rolls from TTC and Andrew Knight from Applitools. Editing by Marii Boyken.

Last week, I hosted a webinar with Chris Rolls from TTC. In the webinar, Chris and I talked about the current state of software testing, where it’s going, and how visual AI and automation will impact the future of software testing. In this article, I’ll be recapping the insights shared from the webinar.

The current state of testing

Software testing is often seen in businesses as a necessary evil at the end of the software development lifecycle to find issues before they reach production. Chris and I strongly agree that software quality is crucial to modern businesses to help to achieve modern businesses’ goals and needs to be thought of throughout the software development lifecycle.

Digital transformation and continuous delivery

The largest and most relevant companies today have embraced digital transformation and technology to run their businesses and meet their customers’ needs. To keep up with digital transformation, you need modern software development practices like DevOps and continuous delivery. DevOps requires continuous testing to be successful, but we’re seeing that reliance on manual testing is the biggest challenge that organizations face when adopting DevOps.

Software testing and quality assurance

The software world is changing, so we need to change how we deliver technology. That requires modern software development approaches, which requires modern software testing and software quality approaches. Thankfully, testing and quality are far more top of mind now than in previous years.

From the audience poll results from our webinar, we see that continuous delivery is here to stay. When asked how often do you deploy changes to production, over 40% of people stated that they deploy either daily or multiple times per day. This wasn’t the case 10 to 20 years ago for most organizations unless technology was the product. Now, daily deployments are pretty common for most organizations.

Low rates of test automation

However, we’re seeing that getting high test automation coverage is still a huge challenge. In the survey, 55% of the respondents automate less than half of their testing.

The numbers may have a bit of sample bias because Applitools users actually automate on average 1.7 times higher than other respondents. The responses align with anecdotal experience that a lot of organizations are still in lower test automation coverage around 20% to 50%.

Testing complexity and the amount of testing needed are going up increasingly, and this shows in our survey as well. More than 50% of the respondents test two languages or more, three browsers or more, three devices or more, and two applications or more.

With two thirds of the respondents saying that one of the hardest parts of testing UIs is that they are constantly changing, traditional automation tools can’t handle testing at that speed scale and complexity.

Using AI in testing

We know that there’s a lot of excitement around AI tools, and the survey shows that.

When asked What parts of the testing process are you supporting or planning on supporting with AI, test offering, test prioritization, test execution, test management, and visual regression we all mentioned. The top two answers among respondents were test execution and visual regression.

It’s important to remember that continuous testing is about more than just test automation.

While test automation is key to the process, you still need to incorporate other software testing and software quality practices like load testing, security, user experience, and accessibility.

The future of testing

What we’re trying to achieve in the future of testing is to support modern software development practices. The best way we’re seeing to do this is to have software testing and software quality more tightly integrated into the process. Let’s talk about what a modern software approach looks like.

Testing sooner and more frequently

To get tighter integration of quality into the process, testing can’t just be an activity at the end of the development lifecycle. Testing has to happen continuously for teams to be able to provide fast feedback across disciplines and ensure a quality product. When this is done, we see increased speed not just of testing, but of overall software development and deployment. We also see reduced costs and increased quality.  More defects are found before production, and we see quicker responses when finding defects in production.

Increasing how much testing is automated

Traditional testing approaches tend to be done mostly manually. Increasing test coverage doesn’t mean manual testing goes away.

Automating your test cases frees up time to do more exploratory testing. Exploratory testing should be assisted by different tools, so AI has a good role to play here. Tools like ChatGPT are useful to brainstorm things like what to test next. Obviously we want to increase test automation coverage at all levels, including unit, API, and UI. Intelligent automated UI tests provide us more information than functional tests alone.

Knowing when to use AI in testing

What does the future of testing with AI look like? It’s a combination of people, processes, and technology. Software testers need to be thinking about what skills we need to have to support these new ways of testing and delivering quality software.

We need to uncover if a use case is better served by AI and machine learning than an algorithmic solution. To do this, we need to ask the following questions:

• What is the pain point that we are trying to solve?
• Are there good algorithmic or heuristic solutions available to address the pain point?
• Are there enterprise ready AI and machine learning solutions that help?
  • If there are solutions, what happens if the AI doesn’t work?
  • If there aren’t solutions yet, how do we position ourselves to be ready for them?

“It’s quite trendy to talk about artificial intelligence, but the reason why we’re partnered with Applitools is that they apply real machine learning and artificial intelligence to a problem that is not well solved by other types of solutions on the market.”

Chris Rolls, CEO, Americas, TTC

Integrating visual AI into test automation

Let’s talk about how we can integrate AI into our testing to get some of those advantages of increased speed and coverage discussed earlier.

What is visual AI?

I like to explain visual AI visually. Do you remember those spot-the-difference pictures we had in our activity books from when we were kids?

As humans, we could sit around and play with this to find the differences manually. But what we want is to be able to find these differences immediately. And that is what visual AI has the power to do.

Even when the images are a little bit skewed or off by a couple pixels here or there, visual AI can pinpoint any differences between one view and another.

Now you might be thinking, Andy, that’s cute, but how’s this technology gonna help me in the real world? Is it just gonna solve little activity book problems? Well, think about all the apps that you would develop – whether web, mobile, desktop, whatever you have – and all the possible ways that you could have visual bugs in your apps.

Here we’ve got three different views from mobile apps. One for Chipotle, one for a bank, and another one for a healthcare provider.

Visual AI test automation

You can see that visual bugs are pervasive and they come in all different shapes and sizes. Sometimes the formatting is off, sometimes a particular word or phrase or title is just nulled out. What’s really pesky is that sometimes you might have overlapping text.

Traditional automation struggles to find these issues because traditional automation usually hinges on text content purely or on particular attributes of elements on a page. So as long as something appears and it’s enactable, most traditional scripts will pass. Even though we as humans visually can inspect and see when something is completely broken and unuseable.

This is where visual AI can help us, because what we can do is we can take snapshots of our app over time and use visual AI to detect when we have visual regressions. Because if, let’s say, one day the title went from being your bank’s name to null, it’ll pick it up right away in your continuous testing.

Visual AI in action

In the webinar, I gave a live demo of automated visual testing using Applitools Eyes. In case you missed the demo, you can check it out here:

The future will be autonomous

So all this really cool stuff is powered by visual AI, a real world application of AI looking at images and being able to find things in them like they were humanized. Now you may think this is really cool, but what’s even cooler is this is just the beginning of what we can do with the power of AI and machine learning in the testing and automation space.

What we’re going to be seeing in the next couple years is a new thing called autonomous testing where not only are we automating our tests, but we’re automating the process of developing and maintaining our tests. The tests are kind of almost writing themselves in a sense. And visual AI is going to be a key part of that, because if testing is interaction and verification, what we want to make autonomous is both interaction and verification. And visual AI has already made verification autonomous. We’re halfway there, folks.

Be sure to check out our upcoming events page for new webinars coming soon!Learn how Applitools Eyes uses AI to catch visual differences between releases while reducing false positives. Happy testing!

Read more

Let the Engineers Speak! Part 5: Audience Q&A

Articles — 01.11.2023

Let the Engineers Speak! Part 4: Changing Frameworks

Articles — 01.05.2023

Let the Engineers Speak! Part 3: Favorite Test Integrations

Articles — 12.28.2022

Let the Engineers Speak! Part 2: The Biggest Challenges

Articles — 12.22.2022

In this final part of our Cypress, Playwright, Selenium, or WebdriverIO? Let The Engineers Speak recap series, we will cover the audience Q&A, sharing the most popular questions from the audience and the answers our experts gave. Be sure to read our previous post.

The experts

I’m Andrew Knight – the Automation Panda – and I moderated this panel. Here were the panelists and the frameworks they represented:

• Gleb Bahmutov (Cypress) – Senior Director of Engineering at Mercari US
• Carter Capocaccia (Cypress) – Senior Engineering Manager – Quality Automation at Hilton
• Tally Barak (Playwright) – Software Architect at YOOBIC
• Steve Hernandez (Selenium) – Software Engineer in Test at Q2
• Jose Morales (WebdriverIO) – Automation Architect at Domino’s

The discussion

ICYMI: Cypress ambassador Filip Hric shared What’s New in Cypress 12, which includes an update on the feature just discussed.

Conclusion

This article concludes our Cypress, Playwright, Selenium, or WebdriverIO? Let The Engineers Speak recap series. We got to hear from engineers at Mercari, YOOBIC, Hilton, Q2, and Domino’s about how their teams build their test automation projects and why they made their framework decisions. Our panelists also shared insights into advantages and disadvantages they’ve encountered in their test automation frameworks. If you missed any previous part of the series, be sure to check them out:

• Let the Engineers Speak! Part 1: Real-World Projects
• Let the Engineers Speak! Part 2: The Biggest Challenges
• Let the Engineers Speak! Part 3: Favorite Test Integrations
• Let the Engineers Speak! Part 4: Changing Frameworks

In part 4 of our Cypress, Playwright, Selenium, or WebdriverIO? Let The Engineers Speak recap series, we will recap the stories our panelists have about changing their test frameworks – or not! Be sure to read our previous post where our panelists talked about their favorite test integrations.

The experts

I’m Andrew Knight – the Automation Panda – and I moderated this panel. Here were the panelists and the frameworks they represented:

• Gleb Bahmutov (Cypress) – Senior Director of Engineering at Mercari US
• Carter Capocaccia (Cypress) – Senior Engineering Manager – Quality Automation at Hilton
• Tally Barak (Playwright) – Software Architect at YOOBIC
• Steve Hernandez (Selenium) – Software Engineer in Test at Q2
• Jose Morales (WebdriverIO) – Automation Architect at Domino’s

The discussion

Audience Q&A

So we had a fantastic conversation amongst each of our panel speakers about the considerations they had as their test projects grew and changed. We found that most of our panelists have had to make changes to their test frameworks used as their projects’ needs changed. In the next article, we’ll cover the most popular questions raised by our audience during the webinar.

In part 3 of our Cypress, Playwright, Selenium, or WebdriverIO? Let the Engineers Speak recap series, we will recap our panelists’ favorite integrations in their test automation projects, as well as share our audience’s favorite integrations. Be sure to read our previous post, where our panelists discussed their biggest challenges in test automation.

The experts

I’m Andrew Knight – the Automation Panda – and I moderated this panel. Here were the panelists and the frameworks they represented:

• Gleb Bahmutov (Cypress) – Senior Director of Engineering at Mercari US
• Carter Capocaccia (Cypress) – Senior Engineering Manager – Quality Automation at Hilton
• Tally Barak (Playwright) – Software Architect at YOOBIC
• Steve Hernandez (Selenium) – Software Engineer in Test at Q2
• Jose Morales (WebdriverIO) – Automation Architect at Domino’s

The discussion

Did anyone ever change their framework?

We heard popular tools mentioned like GitHub Actions and Cucumber, as well as some lesser known tools. A few of our panelists are also using Applitools Eyes in their projects to provide faster test coverage of their frontend. In the next article, we’ll cover the answers these experts gave when we posed a sticky question: “Did you ever consider changing your test framework?”