Bilal Haidar, Author at Automated Visual Testing | Applitools

How Visual UI Testing can speed up DevOps flow [Revisited]

Bilal Haidar — Fri, 11 Dec 2020 20:51:09 +0000

At its core, DevOps means applying best practices to automate the process between development and IT teams. In practice, DevOps requires automating the app builds, running tests to verify source code integrity, and automating the release process until the app is ready for deployment.

With the advent of automated visual UI testing, people went about discovering the best way to integrate Automated Visual Testing in the DevOps workflow to speed up the process.
Bilal Haidar

Continuous Integration / Continuous Delivery (CI/CD) makes up the heart of the DevOps workflow that supports the processes, developers, testers, and clients.

With the advent of automated visual UI testing, people went about discovering the best way to integrate Automated Visual Testing in the DevOps workflow to speed up the process.

In this article, I will share the workflow I use to integrate my source code with a CI/CD service to build the code and run the automated visual UI tests. And, I will show the end result – a full integration between the source code management system, and CI/CD, to report any discrepancies or failures in either.

The other half of this article will demonstrate a practical example of implementing a recommended Visual Testing Workflow.

While doing so, we will be covering topics like CI/CD, Source Code Management systems, and much more.

Let’s start!

Source Code Repository Management System

Where do you store your source code? Many of the external source code repository management (SCRM) systems make your code available to your team wherever they are and whenever they need access. Popular SCRM systems include GitHub, BitBucket, GitLab and Azure Repos. If you don’t already use one or more SCRM systems, I highly recommend you start

SCRM underpins DevOps source management. For example, I can add or modify code and push the changes to the centralized repository. My colleagues elsewhere would do the same thing. Developers joining the team can clone the centralized up-to-date repository to begin coding instantaneously.

SCRM systems have developed best practices and workflows. For instance, the diagram below depicts the workflow of implementing a new feature in your application.

Start by cloning or branching to the dev branch.

Create a new branch name relevant to the feature you are adding.
1. Add your code and commit the changes locally.
2. Push the branch to an SCRM system- GitHub for instance.
Create a Pull Request (PR) based on the new branch.
Invite your team members to review and comment.
Incorporate any changes. If need be, approve and merge the PR.
The branch dev has the new feature and ready to deploy.

I use this typical SCRM Workflow on a daily basis. It’s quite simple and straight to the point, yet it organizes the development process with all team members without losing code or overlapping.

Continuous Integration / Continuous Delivery (CI/CD)

Continuous Integration, or CI for short, is a fully-automated development methodology that seamlessly integrates your daily code changes into a centralized source code management system. CI ensures a high level of code integrity by enforcing your own software development process for any code changes.

When employing CI in your development workflow, creating a new PR in your SCRM system triggers the CI workflow via a webhook more often than not.

The CI workflow is depicted in this diagram:

Start by pushing your branch to the SCRM.
Create a Pull Request.
Allow your colleagues to review and comment.
At the same time, creating a PR triggers a Build Pipeline inside the CI Provider.
You can configure the Build Pipeline to build your code and run the tests. In addition, you can add more tasks to the workflow as you see fit.
Finally, the CI Provider reports the results of building the code and running the tests back to the SCRM.
The results will be displayed on the PR page to signal success or failure.

The CI results will be either a success or failure. In case of a failure, you need to review the PR and fix any issues reported. Otherwise, you can safely merge the PR into your code.

On the other hand, the Continuous Delivery, or CD, automates the process of generating a releasable package from your code. The success results of the CI process triggers a Release Pipeline inside the CD Provider. I won’t be covering CD in this article. To find out more about CD, you can read this detailed guide on CD/CI: What’s Continuous Integration and Continuous Delivery.

Visual Testing Workflow

Automated visual UI testing and CI/CD integrate well together! They complement each other and facilitate the work of DevOps.

Once you have an automated integration and delivery solution, you can easily integrate visual UI tests into your CI/CD workflow. Simply add an additional task in the workflow to run the visual UI tests. Typically, the CI Provider runs the automated tests and sends the results to the Visual Testing Provider. The Visual Testing Provider analyzes the testing results and reports them back to the SCRM on that specific PR.

The diagram below summarizes this whole workflow:

Start by pushing your branch to the SCRM system.
Create a Pull Request.
Let your colleagues review and comment.
At the same time, creating a PR triggers a Build Pipeline inside the CI Provider.
The Build Pipeline builds your code and runs the tests (unit, integration, function and visual UI tests).
The CI Provider reports back the results to the SCRM system.
Meanwhile, the Visual Testing Provider collects the test run results, analyzes them and reports back success or failure to the SCRM system.

Every Visual Testing Provider supports an SCRM system integration to ease communication when running visual UI tests. For instance, Applitools offers GitHub, BitBucket, and other integrations with SCRM systems.

Let’s see this workflow in action in the demo section.

Demo

Now that we know where automated visual UI testing fits in the DevOps workflow, let’s work through an example by trying to automate the entire Visual Testing Workflow.

In this step-by-step guide, we’ll tackle the following:

Cloning the example code.
Configuring Applitools API Key.
Connecting the GitHub repo with CircleCI.
Configuring the GitHub repo.
Connecting the GitHub repo with Applitools.
Running a visual UI test locally.
Running the Visual UI testing workflow.

For this article, I opt to use the source code that I developed for my previous article Cross-browser Testing With Cypress.io. The code contains a single visual UI test that runs against one of the Applitools customer’s website- the https://www.salesforce.com– and fills the contact form located at https://www.salesforce.com/uk/form/contact/contactme/.

1. Cloning the example code

Start by cloning the source code that we’re going to use for this demonstration by running this command:

git clone git@github.com:bhaidar/applitools-github-circle-ci.git

The command clones the GitHub repo to your local computer. Once the code is cloned, enter the applitools-github-circle-ci directory, and run the following command to install the packages:

npm install

This command installs all the necessary NPM packages used by the source code. The app is now ready!

2. Configuring Applitools API Key

Before you can locally run Applitools Eyes Cypress SDK, make sure you get an Applitools API Key, and store it on your machine as an environment variable.

To set the APPLITOOLS_API_KEY environment variable, you can use the export command on Mac or set command on Windows as follows:

View the code on Gist.

3. Connecting the GitHub repo with CircleCI

For this tutorial, we’re going to integrate with our GitHib repo with CircleCI. If you’re not familiar with CircleCI, check out their documentation and sign up for a free account with GitHub or Bitbucket.

Once you’ve created an account and logged in to the CircleCI website, click the Add Projects menu item.

Search and locate your GitHub repo and click the Set up Project button as shown in the diagram.

You are then redirected to the Project Settings page to complete the set up of your project.

On the Project Settings page, select the Linux operating system option and Node as a language. Scroll down the page until you see:

The CircleCI team offers you a step by step guide on how to integrate CircleCI into your application. Make sure you follow the steps listed as shown here:

Create a folder called .circleci and add a file called config.yml
Populate the config.yml file with the sample on the page
Update the sample .yml to reflect your project’s configuration
Push the change to GitHub.
Start building! This will launch your project on CircleCI and its webhooks will listen for updates to your work.

Switch back to your code editor and let’s add the CircleCI requirements.

3a. Build in GitHub

Start by creating the dev git branch by running the commands below:

git checkout -b dev
git push --set-upstream origin dev

Then create a new folder named .circle at the root folder of the app. Then create a new file inside the .circleci folder named config.yml.

The CircleCI provides a sample config.yml file that you can use to start working with CircleCI. For now, paste the following inside the config.yml file:

View the code on Gist.

The “config.yml” defines a single build job. The build job defines some general configurations and some steps.

This job makes uses of the cypress/base:10 Docker image. It also defines the working directory to be the name of the app itself.

The build job steps are:

Checks out the source code to the working directory inside the Docker container.
Restores the cached NPM packages if any.
Installs all NPM package dependencies located in the package.json file.
Stores the packages inside the cache to retrieve and use them later.
Runs a multi-line command as follows:
1. Exports the APPLITOOLS_BATCH_ID to the CIRCLE_SHA1 auto-generated value at runtime.
2. Runs the single visual UI test with Cypress.io

It’s very important to include both commands, exporting the APPLITOOLS_BATCH_ID environment variable, and running the visual UI test, in the same command inside the config.yml file.

3b. Integrate with Applitools

In addition, we need to amend the applitools.config.js file to include the batchId. The applitools.config.js file should now look as follows:

module.exports = {
    showLogs: false,
    batchName: 'Circle CI batch',
    batchId: process.env.APPLITOOLS_BATCH_ID
}

One task remains – adding the APPLITOOLS_API_KEY environment variable on the CircleCI project level, so the visual UI tests can run and connect with Applitools servers. You have the choice to either place this environment variable inside the config.yml file or add the environment variable on the CircleCI website. The latter is best and the recommended option so you don’t share your API on GitHub. You simply embed it at the CircleCI project level.

Locate and click the Jobs menu item on CircleCI.
Near the project name, click the gear icon.
Click the Environment Variables menu item.
Click on the Add Variable button.
On the popup modal window, fill in the following fields:
1. Name: The name of the environment variable in this case APPLITOOLS_API_KEY.
2. Value: Copy the API Key from the Applitools website and paste it here.
Click the Add Variable button to add this environment variable.

3c. Push Changes

Now let’s push these changes to our local copy of source code to GitHub. Automatically, CircleCI will detect a change in the repo and will run the test job. Consequently, the Applitools visual UI test will run.

Run this commands to push your changes to GitHub:

git add .
git commit -m "Add CircleCI support to app"

Switch back to CircleCI and check the status of the job.

The diagram lists all the steps as defined in the config.yml file. CircleCI runs each step in the same order it appears in the config file.

The job fails! Let’s expand the last step to find out why.

If you read through the logs you will notice the following:

CypressError: Timed out retrying: Expected to find element: 'input[id="reg_form_1-UserFirstName"]', but never found it.

Cypress.io has generated an error as it cannot find an input with ID reg_form_1-UserFirstName on the contact form.

The SalesForce team has changed the IDs of the contact form since I wrote the visual test script in my previous article!

Don’t worry, we will fix this in the coming sections. More importantly, we have linked our GitHub repo with CircleCI.

Let’s move on and tackle other tasks.

4. Configuring the GitHub repo

Before we proceed, let’s configure our GitHub repo with the best practices.

Locate the GitHub repo and go to the Settings page on GitHub.
Click on the Branches menu section.
Click the Add rule button to define the approval on new code pushed to this repo.
Enter the name master into the Branch Name Pattern field. The rules we are going to add target only the master branch in this repo. You are free to decide on the branch you want.
Usually, I opt to check the Require pull request reviews before merging option. For the sake of this demonstration, and since I am the only developer working on this code, I will keep it unchecked.
Make sure to check the Require status checks to pass before merging option. This ensures CircleCl allows you to merge a PR into your master branch after a success status outcome.
Check the Require branches to be up to date before merging option, and then select the ci/circleci: build checkbox. In other words, you are specifying the CircleCI job to listen to for status checks.
Locate and click the Create button to add this rule.

That’s it for GitHub!

Let switch to CircleCI and configure one option there.

Locate and click the Jobs menu item on CircleCI.
Click the gear icon on the top-right side of the page.

Locate and click the Advanced Settings menu item.
Scroll down the page until you hit the option Only build pull requests, and make sure to select On. This way, CircleCI build process will be executed only when a new PR is created, and not on every push to the master branch.

That’s it for CircleCI!

5. Connecting the GitHub repo with Applitools

So far, we have managed to integrate CircleCI and GitHub successfully. Any status update on running the build job on CircleCI will be reflected on the GitHub repo.

Let’s now integrate GitHub with Applitools so that our GitHub repo will receive any status updates on whether the visual UI tests ran successfully or failed.

Go to the Applitools Test Manager website and log in with your GitHub credentials.

Navigate to the Admin page.
Locate and click the Teams tile.
In my case, I am the only member of the team. I click the row that represents my own record.
Click the Integrations tab.
Locate and click the GitHub option.
Click the Manage repositories button.
Sign in to your GitHub account again as requested. Click the Sign button and authorize Applitools to access your GitHub account.
Locate your GitHub repository and click Apply.

That’s all! Now any status updates on visual UI tests on the GitHub repo you’ve selected will appear right away.

Now, if you navigate back to the GitHub repo Settings page and click the Webhooks menu item, you will see two Webhooks were added to your repo. One for CircleCI, and another for Applitools.

6. Running a visual UI test locally

In this section, I will be fixing the single visual UI test script that I have by changing the ID names for all the fields on the SalesForce contact form so that the test runs successfully.

As mentioned previously in my articles, the recommended approach for doing visual testing with Cypress.io is to assign data-* attributes to fields. Such data attributes never change, and provide a consistent way to keep your visual tests running.

Since we don’t have control over the source code behind the contact form, I will simply create a new Git branch and fix the visual test code to reflect the new IDs being used.

Run the following commands to create a new branch in Git:

git checkout -b amend-visual-test-to-use-new-ids
git push --set-upstream origin amend-visual-test-to-use-new-ids

I’ll go through the script code and change the way I find input fields on the form:

// Fill First Name
cy.get('input[id="reg_form_1-UserFirstName"]')
.type('John')
      .should('have.value', 'John');

To something like this:

// Fill First Name
cy.get('input[id^="UserFirstName"]')
.type('John')
.should('have.value', 'John');

The new script code now looks like this:

View the code on Gist.

Let’s give it a shot and run the single test, included in the source code, locally, and make sure it runs.

To run the test, we need first to open the Cypress Test Runner app by running this command:

npx cypress open

Click the salesforce-contactus.spec.js file and Cypress will run the test for you. The result is shown below:

The test runs successfully! Let’s make sure the test results are displayed on the Applitools Test Manager website.

Let’s visit the Applitools Test Manager, and verify the results.

Great! Our test runs fine.

7. Running the Visual UI testing workflow

Now that our test runs successfully locally, what we need to do is push our new branch to GitHub, create a new PR and watch how CircleCI and Applitools will be triggered. Their status updates will appear on the PR itself on GitHub.

Run the following commands to push the new branch to Github:

git add .
git commit -m "Fix visual test script"
git push

Navigate to the GitHub repo, and create a new PR.

Notice how the CircleCI build pipeline has started:

ci/circleci:build Pending -- CircleCI is running your tests

Let’s wait for CircleCI to finish running the job and for Applitools to report the results of running the tests on this PR:

Both CircleCI and Applitools report success on running the single visual UI test!

Congratulations!

The ci/circleci:build job ran successfully
The scm/applitools job that compares the git branch to the baseline git branch ran successfully
The tests/applitools job that runs all the visual UI tests ran successfully

Only now you can go ahead and merge the PR into the source code.

Conclusion

This article is just the tip of the iceberg on how Applitools automated visual UI tests can seamlessly integrate with a CI engine to support and facilitate the DevOps workflow.

Now that you have a starting point, I will leave the floor to you. Explore,play and configure! Be brave and try more advanced workflows.

How I ran 100 UI tests in just 20 seconds [Revisited]

Bilal Haidar — Fri, 04 Dec 2020 07:13:55 +0000

From time to time, we revisit blog posts that show the benefit of Applitools across different use cases. This post by Bilal Haidar from 2019 was one of the first outlining the benefits of Applitools Ultrafast Grid.

Recently I’ve put together a bunch of tutorials: an overview of Applitools, how to test Storybook with Angular, how to quickly troubleshoot bugs in React apps, and how to test Vue apps with Cypress.

If you’ve gone through these, you’ll notice that the visual tests complete in seconds — faster than Nic Cage can jack your ride.

How so fast?

The answer is simple: Applitools Ultrafast Grid. Each time you run a visual test in Applitools, Ultrafast Grid renders that page across a broad range of browsers and configurations. In parallel. In seconds.

Applitools Visual AI then grabs a screenshot of each page, compares it to its baseline screenshot, and determines visual differences and the root cause unpinning them. In less than a minute, your cross-browser testing is done and your developers have what they need to fix any visual bugs.

It’s our very own ludicrous mode (a.k.a. how a Tesla can go from zero to 60 mph in a little over two seconds).

But more than a cool technical trick, Ultrafast Grid can help you keep your app development projects on schedule.

In this article, I will demonstrate the Ultrafast Grid system with test runs and a variety of configuration settings, such as multiple browsers, viewports, and devices, to show Applitools’ efficiency in delivering results.

First off, make sure you have the following installed and configured on your machine:

Before we delve into writing code, let’s take a behind-the-scenes look at how Applitools Ultrafast Grid works.

Life before Ultrafast Grid

Before the arrival of the Ultrafast Grid, visual tests were run in this specific order:

A test run checked the pages one by one.
A screenshot was taken for each page.
The screenshot was uploaded to the backend server, where it was analyzed (that is, compared to previous baselines).
Based on the results, the test would either fail or keep running, checking other pages.

So far, so good!

But problems arise when hundreds of screenshots are uploaded to the backend server to be analyzed. Varying parameters ranging from different viewports, different browsers, and different device emulators can also slow things down.

To help solve this, Applitools designed the Ultrafast Grid system to run tests quickly.

Life with Ultrafast Grid

Applitools Ultrafast Grid is split into two components: server-side and client-side. Here’s a diagram of how it works:

The Ultrafast Grid client is a Node JS library, used internally by both Applitools Storybook and Cypress SDKs. Here’s more on the Ultrafast Grid client.

View the code on Gist.

That code snippet instantiates a new Ultrafast Grid client and returns an object. In this case, we are only interested in the openEyes() method returned.

When called, the openEyes() method starts a new test run and sends the Ultrafast Grid backend server details about it– including, but not limited to the application name, batch or test name, the browsers used, and different variations of browser settings to be adapted while running the tests. This method returns a Promise to an object with the following functions: checkWindow() and close().

View the code on Gist.

This method call instructs the Ultrafast Grid backend server to prepare a testing environment for a new test run that requires running the test over two different browsers. The first browser emulates an iPhone X, while the second specifies the desktop viewport to run the test.

To start collecting snapshots, the test issues a call to checkWindow() function. The Ultrafast Grid client handles this call and collects all the needed resources to store the snapshot locally.

At the end of the test, you issue a call to close() method to close and finish the test run.

At this moment, all the DOM Snapshots collected locally during the test run are sent to the Ultrafast Grid backend server. Based on the test run requirements, sent previously with the openEyes() method, the Ultrafast Grid:

Launches a number of browsers with different viewports and device emulators.
Runs all of the DOM snapshots in parallel.
Collects image screenshots.
Sends all the screenshots to the Applitools AI server to do all the testing analysis and returns the results.

Finally, it’s worth mentioning that Ultrafast Grid sits behind a family of Applitools SDKs including the following:

Applitools for Selenium IDE

Now that you understand how the Ultrafast Grid system works so efficiently, let’s throw a spanner in the works by running tests with a variety of browser configuration settings. Then, we’ll review the results on the Applitools Test Manager.

Source code

For this article, I’ve chosen to write a few Storybook stories and use Applitools for the custom Angular ContentEditable component. I’ve described and used this in my previous article on Mixing Storybook with Angular with a sprinkle of Applitools.

There have been some changes to the Applitools Eyes SDK for Storybook npm package since my previous article was published. I’d recommend removing the old package and installing the new one by following these steps:

Step 1: Remove the old @applitools/eyes.storybook NPM package by issuing the following command:

npm remove @applitools/eyes.storybook

copy code to clipboard

Step 2: Install the new @applitools/eyes-storybook NPM package by issuing the following command:

npm i @applitools/eyes-storybook

copy code to clipboard

That’s it!

Applitools Ultrafast Grid Demo

Before writing our tests, let’s run the application and make sure it’s working fine.

Step 1: Clone the application repo.

git clone https://github.com/thisdot/storybook-angular-applitools.git

copy code to clipboard

Step 2: Install all app dependencies by issuing the following command:

Go to the folder /storybook-angular-applitools then run this command:

npm install

copy code to clipboard

Step 3: Run the app by issuing the following command:

ng serve

copy code to clipboard

Now that we are sure the application is up and running switch back to the Terminal and hit Ctrl+c twice. This terminates the running application.

Step 4: Run the Storybook stories by issuing the following command:

npm run storybook

copy code to clipboard

Step 5: Run the Applitools Eyes SDK for Storybook by issuing the following command:

npx eyes-storybook

copy code to clipboard

Make sure you get an Applitool API Key and store it on your machine.

To set the APPLITOOLS_API_KEY environment variable locally on your machine you can use the export command on Mac OS or set command on Windows as follows.

View the code on Gist.

For a complete tutorial on how to install and run Applitools Storybook SDK for Angular, check out this article: Storybook Angular Tutorial.

Back to the command above, this command simply opens the Storybook stories, runs them one by one, and then sends all the DOM snapshots to Applitools Ultrafast Grid, which then renders them.

If you expand the first test run, you get the following results:

Now that the app, Storybook stories, and Applitools are running locally, let’s start playing with our visual tests.

Visual UI tests with custom viewports

Keeping in mind that Applitools with Storybook doesn’t need any kind of configuration, the important test configurations are auto-inferred by Applitools. However, if you want to play around with the test configuration, you have two options to specify:

Environment variables
The applitools.config.js file

For this demonstration, I will make use of the applitools.config.js file. For a full list of test configuration, check the Advanced Configuration section on the @applitools/eyes-storybook repo on Github.

When you’re using Applitools with Storybook, there’s no need to add any specific Applitools commands or set test configuration to enable visual testing.

Applitools Eyes SDK runs the Storybook stories, takes a snapshot of each, and sends the DOM snapshots to the Ultrafast Grid backend server. It saves the test run with the default test configuration.

Step 1: Add a new JavaScript file to the root of your application. Give it a name of applitools.config.js.

Step 2: Add the appName and batchName configuration settings as follows:

View the code on Gist.

Step 3: Run the test case on two different browsers with two different viewports by adding the following:

View the code on Gist.

Consequently, Ultrafast Grid launches a set of instances for each of the browsers and runs the tests simultaneously. The first time these tests are run, we create baseline images to compare against during subsequent test runs.

Step 4: Save and run the tests again by issuing the following command:

npx eyes-storybook

copy code to clipboard

Step 5: Check the test run in Applitools Test Manager.

I’ve switched the Test Manager view to the Batch Summary View.

Notice how Applitools Ultrafast Grid ran your tests twice, as per the test configuration file above. The test configuration above specified two different browsers with two different viewports. This means that the Ultrafast Grid runs every test twice– once per browser configuration set.

Running this test yields an execution time of just four seconds! To be clear, this is what we spent rendering baseline images, once the DOM snapshots were uploaded. Even so, that’s really fast.

It’s the duty of the Ultrafast Grid to analyze the test configuration file attached to the test and spawn, in parallel, a number of browser instances to handle the test run efficiently.

Visual UI tests with device emulation

Let’s run another test. This time, instead of specifying the browser viewports, we will specify a device emulation for the browser. Device emulation uses a Chrome browser, with the viewport dimensions, and user agent string, specific to that device.

Step 1: Replace the content of the applitools.config.js file with the following configuration:

View the code on Gist.

In this test configuration, we are specifying to the Ultrafast Grid to run this test over a Chrome browser with a device emulation of an iPhone X using a landscape screen orientation.

Ultrafast Grid supports only the Chrome browser for running tests over a device emulation, so you can skip the name of the browser. I’m explicitly specifying the browser name to make it very clear that this is a browser emulation and not a real device.

For a complete list of device emulators supported by the Chrome browser, take a look at this page: Chromium Device Emulators.

Step 2: Save and run the tests by issuing the following command:

npx eyes-storybook

copy code to clipboard

Step 3: Check the test run in Applitools Test Manager.

I’ve switched the Test Manager view to the Batch Summary View.

As per the new test configuration file, the Ultrafast Grid runs your tests on a Chrome browser with a device emulation for iPhone X. Hence, the browser name shown is Safari 11.0 and landscape with a viewport of 812×375.

Notice that running this test yields an execution time of just two seconds! Again, this is once the DOM snapshot has been uploaded. But still…

Finally, in Applitools, baseline snapshots are stored as per test configuration settings. Different test configuration settings generate different baseline snapshots. You can’t compare apples to oranges, right?

That’s why, if you click on the snapshot above (after switching back to Batch steps view) and show both the baseline and the current test run, you won’t get any baseline snapshot–because it’s the first time we’re running the test with these configuration settings.

100 Visual UI tests

Let’s go crazy and run 100 Visual/UI tests with a combination of test configurations.

Step 1: Replace the content of the applitools.config.js file with the following configuration:

View the code on Gist.

In this test configuration, we are running 100 tests with 50 unique view ports on two major browsers: Firefox and Google Chrome.

Step 2: Save and run the tests by issuing the following command:

npx eyes-storybook

copy code to clipboard

During this time, the Eyes Storybook SDK will read all Storybook stories in the application, run them one by one using the Storybook engine, generate a DOM snapshot for each test, and finally upload the results to the Eyes server. At this moment, the Eyes server will run each and every DOM snapshot, render an image snapshot, and store it as a baseline snapshot.

The Eyes SDK/Server took only 21 seconds to complete all the above tasks, create baselines, and report the results on Applitools Test Manager Dashboard, for the entire batch of 100 test runs for a single Storybook story.

Step 3: Check the test run in Applitools Test Manager.

I’ve switched the Test Manager view to the Batch Summary View.

Let’s see the power of the Eyes server when running a regression testing cycle.

Step 4: Let’s introduce a visual change on the Storybook story and run the Eyes SDK once again.

Replace the content of the /src/stories/index.stories.js file with the following:

View the code on Gist.

view raw regression-testing-cycle hosted with by GitHub

Now the ContentEditableComponent should render with a Blue background.

Step 5: Run the tests again by issuing the following command:

npx eyes-storybook

copy code to clipboard

Once again, the Eyes SDK/Server took only 21 seconds to upload the DOM snapshot, run the entire batch of visual UI tests, and display the results on the Applitools Test Manager Dashboard.

Notice the error message generated saying that “A total of 100 differences were found”.

Because the change we introduced affected the entire batch of test runs, therefore, all the current tests failed in comparison with the baseline snapshots, due to the visual change introduced earlier.

Step 6: Check the test run in Applitools Test Manager.

The Test Manager highlights the differences between the baseline snapshot and the current test run and displays a mismatch icon near the snapshots.

At this moment, you can either accept the new changes and make them the new baseline or reject them and keep the old baseline.

Once again, it takes the power of the Ultrafast Grid to analyze the test configuration file and set up the testing environment as needed.

See for yourself

Here’s a step-by-step tutorial video of how I did all the steps above. You can see how it took just 20 seconds for the Applitools Storybook SDK to upload all the DOM snapshots, then use Applitools Visual AI to look for visual regressions:

Conclusion

By now, you have some insight into how Applitools Ultrafast Grid functions behind the scenes when running your visual tests.

Want to get run your own tests ludicrously fast? Applitools Ultrafast grid is GA for Storybook, Cypress, and Selenium IDE. If you use Selenium WebDriver with Java, JavaScript, C#, Python, or Ruby, request to join our early access program.

Happy Testing!

Photo by Sander Jeurissen on Unsplash

The post How I ran 100 UI tests in just 20 seconds [Revisited] appeared first on Automated Visual Testing | Applitools.

Visual Testing with Applitools, Appium, and Amazon AWS Device Farm

Bilal Haidar — Thu, 18 Jun 2020 17:28:18 +0000

Visual UI testing is more than just testing your app on Desktop browsers and Mobile emulators. In fact, you can do more with Visual UI testing to run your tests over physical mobile devices.

Visual UI testing compares the visually-rendered output of an application against itself in older iterations. Users call this type of test version checking. Some users apply visual testing for cross-browser tests. They run the same software version across different target devices/operating systems/browsers/viewports. For either purpose, we need a testing solution that has high accuracy, speed, and works with a range of browsers and devices. For these reasons, we chose Applitools.

Running your Visual UI testing across physical devices means having to set up your own local environment to run the tests. Imagine the number of devices, screen resolutions, operating systems, and computers you’d need! It would be frustratingly boring, expensive, and extremely time-consuming.

This is where Amazon’s AWS Device Farm comes into play. This powerful service can build a testing environment. It uses physical mobile devices to run your tests! All you do is upload your tests to Amazon, specify the devices you want, and it will take it from there!

In one of my recent articles, How Visual UI Testing can speed up DevOps flow I showed how you can configure a CD/CI service to run your Visual UI tests. The end result would be the same, whether you are running your tests locally, or via such services. Once the tests run, you can always check the results over the Applitools Test Manager Dashboard.

In this article, I will show you how you can run your Visual UI tests, whether you’ve written them for your mobile or web app, on real physical mobile devices in the cloud. For this, I will be employing Applitools, Appium, and AWS Device Farm.

AWS Device Farm for mobile visual testing

AWS Device Farm is a mobile app testing platform that helps developers automatically test their apps on hundreds of real devices in minutes.

When it comes to testing your app over mobile devices, the choices are numerous. Amazon helps to build a “Device Farm” on behalf of the developers and testers, hence the name.

Here are some of the major advantages and features for using this service:

Cross-platform. Android and iOS platforms (Native, Hybrid, and Web) are all supported. This includes native apps built with:
- Java/Kotlin for Android.
- Swift for iOS.
- PhoneGap.
- Xamarin.
- Unity.
- And web apps built for mobile browsers.

Scale. AWS Device Farm supports hundreds of unique physical devices, categorized by make, model, and operating system. You may also choose to run your tests across multiple instances of the same device. All these devices are available for you with a few mouse clicks!

Safety and Security. AWS Device Farm provides full hardware and software isolation. The devices are physically isolated from one another! They cannot feed each other, so there’s no way for one phone to take a photo, video, or audio recording of a device sitting next to it. In addition, the devices are not visible to each other from a wireless or network point of view. Bluetooth and Wi-Fi traffic is not shared. On the software side, the devices are dynamically tethered to a host machine. When you run your test on a host machine, it has the device plugged into it over USB. That very host machine, that executes your code, is brought up on the fly, runs your code, and is then torn down. It’s never reused between customers.

Reporting. The test results, together with any screenshots, videos, logs and performance logs, are all logged, and saved in the cloud. The AWS Device Farm offers a rich Dashboard to allow you to browse any of these logs in order to debug your test runs.

AWS Device Farm supports a number of test runners. This includes Appium Java JUnit, Appium Python, Appium Ruby, and Appium Java TestNG. Back in January 2019, Amazon announced support for the Appium Node.js test runner. This means you can build your tests with Selenium Webdriver, for instance, and have it run on top of AWS Device Farm.

Now that you have an idea about AWS Device Farm, let’s move on, and discover the Appium automation testing framework.

Selenium Webdriver for browser app automation

Selenium WebDriver is a browser automation framework that allows a developer to write commands, and send them to the browser. It offers a set of clients with a variety of programming languages (Java, JavaScript, Ruby, Python, PHP and others).

Figure 1 below shows the Selenium WebDriver architecture:

Figure 1: Selenium WebDriver Architecture

Selenium WebDriver architecture consists of:

Selenium Language Bindings. These bindings are Selenium Client Libraries, offered in multiple programming languages, that developers use to send control commands to the browser. For instance, a developer can open a browser instance, and query for an element in DOM, among other tasks.

JSON Wire Protocol. This is a REST API Protocol (JSONWP) that all WebDriver Server implementations adhere to and understand. Each of the queries and commands the developers write using the Selenium Client Library are converted to HTTP Requests, with the query or command, as payload in the JavaScript Object Notation (JSON) format, and is sent to the WebDriver Server.

Browser Drivers. These are WebDriver Server implementations for a variety of browsers. A WebDriver Server implementation is nothing but an HTTP Server that receives requests from the Selenium Client Library using the JSON Wire Protocol format. It then analyzes the HTTP Request and prepares a browser-specific command to execute against the browser. If the request is a GET request, the browser-driver should return a response. Otherwise, a POST request is a one-way request to execute an action only without any response.

Browsers. These are the browsers that have a corresponding WebDriver Server implementation. The WebDriver Server implementations communicate with the Browsers over HTTP via the DevTools Protocol of each browser.

Selenium 4 is obseleting the JSONWP in favor of the new W3C WebDriver standard.

Here’s a quick tutorial on using and learning Selenium WebDriver.

With that brief overview of Selenium WebDriver, let’s move on and explore Appium.

Appium for mobile app automation

Appium is an open-source tool to automate Mobile app testing. It’s a cross-platform that supports both OS (Android and iOS) test scripts. It is tested on simulators (iOS), emulators (Android) and real devices (iOS, Android).

It’s an HTTP Server written in Node.js that creates and handles WebDriver sessions. When you install Appium, you are actually installing the Appium Server. It follows the same approach as the Selenium WebDriver, which receives HTTP requests from the Client Libraries in JSON format with the help of JSONWP. It then handles those HTTP Requests in different ways. That’s why you can make use of Selenium WebDriver language bindings, client libraries and infrastructure to connect to the Appium Server.

Instead of connecting a Selenium WebDriver to a specific browser WebDriver, you will be connecting it to the Appium Server. Appium uses an extension of the JSONWP called the Mobile JSON Wire Protocol (MJSONWP) to support the automation of testing for native and hybrid mobile apps.

It supports the same Selenium WebDriver clients with a variety of multiple programming languages such as Java, JavaScript, Ruby, Python, PHP and others.

Being a Node.js HTTP Server, it works in a client-server architecture. Figure 2 below depicts the Appium Client-Server Architecture model:

Figure 2: Appium Server Architecture

Appium architecture consists of:

Appium Client. The Client Library that communicates with the Appium Server via a session by sending the commands and queries over HTTP in JSON format. These requests are eventually executed against the specific Mobile device (emulator or real device).

Mobile JSONWP. This is the communication Protocol that both Appium Clients and Appium Server understand and use to pass along the commands and queries to be executed. The Appium Server differentiates between iOS and Android Request using the Desired Capabilities argument. It’s a collection of keys and values encoded in a JSON object, sent by Appium clients to the server when a new automation session is requested. They contain all the information about the device to be used to run the tests against. Here’s a detailed tutorial on all possible desired capabilities to use with Appium. Desired Capabilities for Appium

Appium Server. It’s a Node.js HTTP server that receives requests from the client libraries, translates them into meaningful commands and passes them over to the specific UI Automator.

UI Automators. They are used to execute commands against the Mobile device/emulator/simulator. Examples are: UI Automator2 and XCUITest (iOS)

The results of the test session are then communicated back to the Appium Server, and back to the Client in the form of logs, using the Mobile JSONWP.

Now that you are well equipped with knowledge for Selenium WebDriver and Appium, let’s go to the demo section of this article.

Demo

In this section, we will write a Visual UI test script to test a Web page. We will run the tests over an Android device both locally and on AWS Device Farm.

I will be using both Selenium WebDeriver and Appium to write the test script.

Prerequisites

Before you can start writing and running the test script, you have to make sure you have the following components installed and ready to be used on your computer:

Java installed
JAVA_HOME environment variable is set to the Java SDK path
Node.js installed

Assuming you are working on a MacOS computer, you can verify the above installations by running the following bash commands:

echo $JAVA_HOME // this should print the Java SDK path
node -v // this should print the version of Node.js installed
npm -v // this should print the version of the Node Package Manager installed

Component Installations

For this demo we need to install Appium Server, Android Studio / SDK and finally make sure to have a few environment variables properly set.

Let’s start by installing Appium Server. Run the following command to install Appium Server locally on your computer.

npm install -g appium

The command installs the Appium NPM package globally on your computer. To verify the installation, run the command:

appium -v // this should print the Appium version

Now let’s install Android Studio / SDK so that you can run the test script on an emulator or real device. You could install the Android SDK only but then you have to do additional advanced steps to properly configure the Android environment on your computer. I highly recommend installing the Android Studio as it makes your life easier.

Download the Android Studio executable. Follow the steps below to install locally on your computer:

Notice the location where the Android SDK was installed. It’s /Users/{User Account}/Library/Android/sdk.

Wait until the download and installation is complete. That’s all!

Because I want to run the test script locally over an Android emulator, let’s add one.

Open the Android Studio app:

Click the Configure icon:

Select the AVD Manager menu item.

Click the + Create Virtual Device button.

Locate and click the Pixel XL device then hit Next.

Locate the Q release and click the Download link.

Read and accept the Terms and Conditions then hit Next.

The Android 10, also known as Q release, starts downloading.

Once the installation is complete, click the Next button to continue setting up an Android device emulator.

The installation is complete. Grab the AVD Name as you will use it later on in the test script, and hit Finish.

Finally, we need to make sure the following environment variables are set on your computer. Open the ~/.bash_profile file, and add the following environment variables:

APPLITOOLS_API_KEY={Get the Applitools API Key from Applitools Test Manager}
export APPLITOOLS_API_KEY
ANDROID_HOME=/Users/{Use your account name here}/Library/Android/sdk
export ANDROID_HOME
ANDROID_HOME_TOOLS=$ANDROID_HOME/tools
export ANDROID_HOME_TOOLS
ANDROID_HOME_TOOLS_BIN=$ANDROID_HOME_TOOLS/bin
export ANDROID_HOME_TOOLS_BIN
ANDROID_HOME_PLATFORM=$ANDROID_HOME/platform-tools
export ANDROID_HOME_PLATFORM
APPIUM_ENV="Local"
export APPIUM_ENV

Finally, add the above environment variables to the $PATH as follows:

export $PATH=$PATH:$ANDROID_HOME:$ANDROID_HOME_TOOLS:$ANDROID_HOME_TOOLS_BIN:$ANDROID_HOME_PLATFORM

One last major component that you need to download, and have on your computer, is the ChromeDriver. Navigate to the Appium ChromeDriver website, and download the latest workable ChromeDriver release for Appium. Once downloaded, make sure to move the file to the location: /usr/local/bin/chromedriver

That’s it for the installations! Let’s move on and explore the Visual UI test script in depth.

Run the Visual UI Test Script locally

You can find the source code demo of this article on this GitHub repo.

Let’s explore the main test script in this repo.

"use strict";

;(async () => {

    const webdriver = require("selenium-webdriver");
    const LOCAL_APPIUM = "https://web.archive.org/web/20221206000829/http://127.0.0.1:4723/wd/hub";

    // Initialize the eyes SDK and set your private API key.
    const { Eyes, Target, FileLogHandler, BatchInfo, StitchMode } = require("@applitools/eyes-selenium");

    const batchInfo = new BatchInfo("AWS Device Farm");
    batchInfo.id = process.env.BATCH_ID
    batchInfo.setSequenceName('AWS Device Farm Batches');
    
    // Initialize the eyes SDK
    let eyes = new Eyes();
    eyes.setApiKey(process.env.APPLITOOLS_API_KEY);
    eyes.setLogHandler(new FileLogHandler(true));
    eyes.setForceFullPageScreenshot(true)
    eyes.setStitchMode(StitchMode.CSS)
    eyes.setHideScrollbars(true)
    eyes.setBatch(batchInfo);

    const capabilities = {
        platformName: "Android",
        deviceName: "Android Emulator",
        automationName: "UiAutomator2",
        browserName: 'Chrome',
        waitforTimeout: 30000,
        commandTimeout: 30000,
    };

    if (process.env.APPIUM_ENV === "Local") {
        capabilities["avd"] = 'Pixel_XL_API_29';
    }
    
    // Open browser.
    let driver = new webdriver
        .Builder()
        .usingServer(LOCAL_APPIUM)
        .withCapabilities(capabilities)
        .build();

    try {
        // Start the test
        await eyes.open(driver, 'Vuejs.org Conferences', 'Appium on Android');

        await driver.get('https://us.vuejs.org/');

        // Visual checkpoint #1.
        await eyes.check('Home Page', Target.window());

        // display title of the page
        await driver.getTitle().then(function (title) {
            console.log("Title: ", title);
        });

        // locate and click the burger button
        await driver.wait(webdriver.until.elementLocated(webdriver.By.tagName('button.navbar__burger')), 2000).click();
        
        // locate and click the hyperlink with href='/#location' inside the second nav element
        await driver.wait(webdriver.until.elementLocated(webdriver.By.xpath("//web.archive.org/web/20221206000829/https://nav[2]/ul/li[3]/a[contains(text(), 'Location')]")), 2000).click();

        const h2 = await driver.wait(webdriver.until.elementLocated(webdriver.By.xpath("(//h2[@class='section-title'])[4]")), 2000);
        console.log("H2 Text: ", await h2.getText());

        // Visual checkpoint #2.
        await eyes.check('Home Loans', Target.window());

        // Close Eyes
        await eyes.close();
    } catch (error) {
        console.log(error);
    } finally {
        // Close the browser.
        await driver.quit();

        // If the test was aborted before eyes.close was called, ends the test as aborted.
        await eyes.abort();
    }
})();

The test script starts by importing the selenium-webdriver NPM package.

It imports a bunch of objects from the @applitools/eyes-selenium NPM package.

It constructs a BatchInfo object used by Applitools API.

const batchInfo = new BatchInfo("AWS Device Farm");
batchInfo.id = process.env.BATCH_ID
batchInfo.setSequenceName('AWS Device Farm Batches');

It then creates the Eyes object that we will use to interact with the Applitools API.

// Initialize the eyes SDK
let eyes = new Eyes();
eyes.setApiKey(process.env.APPLITOOLS_API_KEY);
eyes.setLogHandler(new FileLogHandler(true));
eyes.setForceFullPageScreenshot(true)
eyes.setStitchMode(StitchMode.CSS)
eyes.setHideScrollbars(true)
eyes.setBatch(batchInfo);

It’s so important to set the Applitools API key at this stage. Otherwise, you won’t be able to run this test. The code above also directs the Applitools API logs to a File located at the root of the project under the name of eyes.log.

Next, we define the device capabilities that we are going to send to Appium.

const capabilities = {
        platformName: "Android",
        deviceName: "Android Emulator",
        automationName: "UiAutomator2",
        browserName: 'Chrome',
        waitforTimeout: 30000,
        commandTimeout: 30000,
    };
    if (process.env.APPIUM_ENV === "Local") {
        capabilities["avd"] = 'Pixel_XL_API_29';
    }

We are using an Android emulator to run our test script over a Chrome browser with the help of the UIAutomator 2 library.

We need to set the avd capability only when running this test script locally. For this property, grab the AVD ID of the Android Device Emulator we set above.

Now, we create and build a new WebDriver object by specifying the Appium Server local URL and the device capabilities as:

const LOCAL_APPIUM = "http://127.0.0.1:4723/wd/hub";
let driver = new webdriver
        .Builder()
        .usingServer(LOCAL_APPIUM)
        .withCapabilities(capabilities)
        .build();

Appium is configured to listen on Port 4723 under the path of /wd/hub.

The rest of the script is usual Applitools business. In brief, the script:

Opens a new Applitools test session
Sends a command to navigate the browser to https://us.vuejs.org/
Grabs the page title and displays it on screen.
Clicks the Burger Button to expand the menu on a Mobile device.
Finds the Location section on the page.
Finally, it prints the H2 text of the Location section.

Notice that the script asserts two Eyes SDK Snapshots. The first captures the home page of the website, while the second captures the Location section.

Finally, some important cleanup is happening to close the WebDriver and Eyes SDK sessions.

Open the package.json file, and locate the two scripts there:

"appium": "appium --chromedriver-executable /usr/local/bin/chromedriver --log ./appium.log",
"test": "node appium.js"

The first runs and starts the Appium Server, and the second to run the test script.

Let’s first run the Appium server by issuing this command:

npm run-script appium

Then, once Appium is running, let’s run the test script by issuing this command:

npm run-script test

Verify Test Results on Applitools Test Manager

You will see the following test results:

The two snapshots have been recorded!

Run the Visual UI Test Script on AWS Device Farm

Now that the test runs locally, let’s run it on AWS Device Farm. Start by creating a new account on Amazon Web Service website.

Create a new project by following the steps below:

Select the Mobile Device Project and name your project Appium.
Click the Create project button.

Locate and click the + Create a new run button.

Select the HTML5 option since we are testing a Web Page on a mobile device.
Assign your test run a name.
Click the Next step button.

Select the Appium Node.js test runner
Upload your tests packaged in a zip file.

Let’s package our app in a zip file in order to upload it at this step.

Switch back to the code editor, open a command window, and run the following:

npm install

This command is essential to make sure all the NPM package dependencies for this app are installed.

npm install -g npm-bundle

The command above installs the npm-bundle NPM package globally on your machine.

Then, run the command to package and bundle your app:

npm-bundle

The command bundles and packages your app files, and folders, including the node_modules folder.

The output of this step creates the file with the .tgz extension.

The final step before uploading is to compress the file by running the command:

zip -r appium-aws.zip *.tgz

Name the file whatever you wish.

Now you can upload the .zip file to AWS Device Farm.

Once the file uploads, scroll down the page to edit the .yaml file of this test run like so:

Make sure you insert your Applitools API Key as shown in the diagram.
Add the node appium.js command to run your test script.
Click the Save Testspec file.

You can name it anything you want.

It’s time to select the devices that you want to run the test script against. I will pick up a customized list of devices. Therefore, click the Create a new device pool button.

Give this new pool a name.
Pick up the selected Android Devices. You may select others too.
Click the Save device pool button.

Now, you can see the new device pool selected with the devices listed.
Click the Next step button.

Locate and click the Confirm and start run button.

The test run starts!
Select the test run listed.

You can watch the progress on both devices as the test is running. Usually, this step takes a bit of time to run and complete.

Finally, the results are displayed clearly, and in our case, all the green indicates a pass.

Verify Test Results on Applitools Test Manager

Switch back to the Applitools Test Manager, and verify the results of this second run via AWS Device Farm.

As expected, we get exactly the same results as running the test script locally.

Conclusion

Given the massive integrations that Applitools offers with its rich SDKs, we saw how easily and quickly we can run our Visual UI tests in the cloud using the AWS Device Farm service. This service, and similar services, enrich the Visual regression testing ecosystem, and make perfect sense when performing them.

For More Information

The post Visual Testing with Applitools, Appium, and Amazon AWS Device Farm appeared first on Automated Visual Testing | Applitools.

How Visual UI Testing can speed up DevOps flow

Bilal Haidar — Fri, 07 Feb 2020 18:05:35 +0000

Continuous Integration / Continuous Delivery (CI/CD) makes up the heart of the DevOps workflow that supports the processes, developers, testers, and clients.

With the advent of automated visual UI testing, people went about discovering the best way to integrate Automated Visual Testing in the DevOps workflow to speed up the process.

The other half of this article will demonstrate a practical example of implementing a recommended Visual Testing Workflow.

While doing so, we will be covering topics like CI/CD, Source Code Management systems, and much more.

Let’s start!

Source Code Repository Management System

SCRM systems have developed best practices and workflows. For instance, the diagram below depicts the workflow of implementing a new feature in your application.

Start by cloning or branching to the dev branch.

Create a new branch name relevant to the feature you are adding.
1. Add your code and commit the changes locally.
2. Push the branch to an SCRM system- GitHub for instance.
Create a Pull Request (PR) based on the new branch.
Invite your team members to review and comment.
Incorporate any changes. If need be, approve and merge the PR.
The branch dev has the new feature and ready to deploy.

I use this typical SCRM Workflow on a daily basis. It’s quite simple and straight to the point, yet it organizes the development process with all team members without losing code or overlapping.

Continuous Integration / Continuous Delivery (CI/CD)

When employing CI in your development workflow, creating a new PR in your SCRM system triggers the CI workflow via a webhook more often than not.

The CI workflow is depicted in this diagram:

Start by pushing your branch to the SCRM.
Create a Pull Request.
Allow your colleagues to review and comment.
At the same time, creating a PR triggers a Build Pipeline inside the CI Provider.
You can configure the Build Pipeline to build your code and run the tests. In addition, you can add more tasks to the workflow as you see fit.
Finally, the CI Provider reports the results of building the code and running the tests back to the SCRM.
The results will be displayed on the PR page to signal success or failure.

The CI results will be either a success or failure. In case of a failure, you need to review the PR and fix any issues reported. Otherwise, you can safely merge the PR into your code.

Visual Testing Workflow

Automated visual UI testing and CI/CD integrate well together! They complement each other and facilitate the work of DevOps.

The diagram below summarizes this whole workflow:

Start by pushing your branch to the SCRM system.
Create a Pull Request.
Let your colleagues review and comment.
At the same time, creating a PR triggers a Build Pipeline inside the CI Provider.
The Build Pipeline builds your code and runs the tests (unit, integration, function and visual UI tests).
The CI Provider reports back the results to the SCRM system.
Meanwhile, the Visual Testing Provider collects the test run results, analyzes them and reports back success or failure to the SCRM system.

Let’s see this workflow in action in the demo section.

Demo

Now that we know where automated visual UI testing fits in the DevOps workflow, let’s work through an example by trying to automate the entire Visual Testing Workflow.

In this step-by-step guide, we’ll tackle the following:

Cloning the example code.
Configuring Applitools API Key.
Connecting the GitHub repo with CircleCI.
Configuring the GitHub repo.
Connecting the GitHub repo with Applitools.
Running a visual UI test locally.
Running the Visual UI testing workflow.

1. Cloning the example code

Start by cloning the source code that we’re going to use for this demonstration by running this command:

git clone git@github.com:bhaidar/applitools-github-circle-ci.git

The command clones the GitHub repo to your local computer. Once the code is cloned, enter the applitools-github-circle-ci directory, and run the following command to install the packages:

npm install

This command installs all the necessary NPM packages used by the source code. The app is now ready!

2. Configuring Applitools API Key

Before you can locally run Applitools Eyes Cypress SDK, make sure you get an Applitools API Key, and store it on your machine as an environment variable.

To set the APPLITOOLS_API_KEY environment variable, you can use the export command on Mac or set command on Windows as follows:

View the code on Gist.

3. Connecting the GitHub repo with CircleCI

Once you’ve created an account and logged in to the CircleCI website, click the Add Projects menu item.

Search and locate your GitHub repo and click the Set up Project button as shown in the diagram.

You are then redirected to the Project Settings page to complete the set up of your project.

On the Project Settings page, select the Linux operating system option and Node as a language. Scroll down the page until you see:

The CircleCI team offers you a step by step guide on how to integrate CircleCI into your application. Make sure you follow the steps listed as shown here:

Create a folder called .circleci and add a file called config.yml
Populate the config.yml file with the sample on the page
Update the sample .yml to reflect your project’s configuration
Push the change to GitHub.
Start building! This will launch your project on CircleCI and its webhooks will listen for updates to your work.

Switch back to your code editor and let’s add the CircleCI requirements.

3a. Build in GitHub

Start by creating the dev git branch by running the commands below:

git checkout -b dev
git push --set-upstream origin dev

Then create a new folder named .circle at the root folder of the app. Then create a new file inside the .circleci folder named config.yml.

The CircleCI provides a sample config.yml file that you can use to start working with CircleCI. For now, paste the following inside the config.yml file:

View the code on Gist.

The “config.yml” defines a single build job. The build job defines some general configurations and some steps.

This job makes uses of the cypress/base:10 Docker image. It also defines the working directory to be the name of the app itself.

The build job steps are:

Checks out the source code to the working directory inside the Docker container.
Restores the cached NPM packages if any.
Installs all NPM package dependencies located in the package.json file.
Stores the packages inside the cache to retrieve and use them later.
Runs a multi-line command as follows:
1. Exports the APPLITOOLS_BATCH_ID to the CIRCLE_SHA1 auto-generated value at runtime.
2. Runs the single visual UI test with Cypress.io

It’s very important to include both commands, exporting the APPLITOOLS_BATCH_ID environment variable, and running the visual UI test, in the same command inside the config.yml file.

3b. Integrate with Applitools

In addition, we need to amend the applitools.config.js file to include the batchId. The applitools.config.js file should now look as follows:

module.exports = {
    showLogs: false,
    batchName: 'Circle CI batch',
    batchId: process.env.APPLITOOLS_BATCH_ID
}

Locate and click the Jobs menu item on CircleCI.
Near the project name, click the gear icon.
Click the Environment Variables menu item.
Click on the Add Variable button.
On the popup modal window, fill in the following fields:
1. Name: The name of the environment variable in this case APPLITOOLS_API_KEY.
2. Value: Copy the API Key from the Applitools website and paste it here.
Click the Add Variable button to add this environment variable.

3c. Push Changes

Run this commands to push your changes to GitHub:

git add .
git commit -m "Add CircleCI support to app"

Switch back to CircleCI and check the status of the job.

The diagram lists all the steps as defined in the config.yml file. CircleCI runs each step in the same order it appears in the config file.

The job fails! Let’s expand the last step to find out why.

If you read through the logs you will notice the following:

CypressError: Timed out retrying: Expected to find element: 'input[id="reg_form_1-UserFirstName"]', but never found it.

Cypress.io has generated an error as it cannot find an input with ID reg_form_1-UserFirstName on the contact form.

The SalesForce team has changed the IDs of the contact form since I wrote the visual test script in my previous article!

Don’t worry, we will fix this in the coming sections. More importantly, we have linked our GitHub repo with CircleCI.

Let’s move on and tackle other tasks.

4. Configuring the GitHub repo

Before we proceed, let’s configure our GitHub repo with the best practices.

Locate the GitHub repo and go to the Settings page on GitHub.
Click on the Branches menu section.
Click the Add rule button to define the approval on new code pushed to this repo.
Enter the name master into the Branch Name Pattern field. The rules we are going to add target only the master branch in this repo. You are free to decide on the branch you want.
Usually, I opt to check the Require pull request reviews before merging option. For the sake of this demonstration, and since I am the only developer working on this code, I will keep it unchecked.
Make sure to check the Require status checks to pass before merging option. This ensures CircleCl allows you to merge a PR into your master branch after a success status outcome.
Check the Require branches to be up to date before merging option, and then select the ci/circleci: build checkbox. In other words, you are specifying the CircleCI job to listen to for status checks.
Locate and click the Create button to add this rule.

That’s it for GitHub!

Let switch to CircleCI and configure one option there.

Locate and click the Jobs menu item on CircleCI.
Click the gear icon on the top-right side of the page.

Locate and click the Advanced Settings menu item.
Scroll down the page until you hit the option Only build pull requests, and make sure to select On. This way, CircleCI build process will be executed only when a new PR is created, and not on every push to the master branch.

That’s it for CircleCI!

5. Connecting the GitHub repo with Applitools

So far, we have managed to integrate CircleCI and GitHub successfully. Any status update on running the build job on CircleCI will be reflected on the GitHub repo.

Let’s now integrate GitHub with Applitools so that our GitHub repo will receive any status updates on whether the visual UI tests ran successfully or failed.

Go to the Applitools Test Manager website and log in with your GitHub credentials.

Navigate to the Admin page.
Locate and click the Teams tile.
In my case, I am the only member of the team. I click the row that represents my own record.
Click the Integrations tab.
Locate and click the GitHub option.
Click the Manage repositories button.
Sign in to your GitHub account again as requested. Click the Sign button and authorize Applitools to access your GitHub account.
Locate your GitHub repository and click Apply.

That’s all! Now any status updates on visual UI tests on the GitHub repo you’ve selected will appear right away.

Now, if you navigate back to the GitHub repo Settings page and click the Webhooks menu item, you will see two Webhooks were added to your repo. One for CircleCI, and another for Applitools.

6. Running a visual UI test locally

In this section, I will be fixing the single visual UI test script that I have by changing the ID names for all the fields on the SalesForce contact form so that the test runs successfully.

Since we don’t have control over the source code behind the contact form, I will simply create a new Git branch and fix the visual test code to reflect the new IDs being used.

Run the following commands to create a new branch in Git:

git checkout -b amend-visual-test-to-use-new-ids
git push --set-upstream origin amend-visual-test-to-use-new-ids

I’ll go through the script code and change the way I find input fields on the form:

// Fill First Name
cy.get('input[id="reg_form_1-UserFirstName"]')
.type('John')
      .should('have.value', 'John');

To something like this:

// Fill First Name
cy.get('input[id^="UserFirstName"]')
.type('John')
.should('have.value', 'John');

The new script code now looks like this:

View the code on Gist.

Let’s give it a shot and run the single test, included in the source code, locally, and make sure it runs.

To run the test, we need first to open the Cypress Test Runner app by running this command:

npx cypress open

Click the salesforce-contactus.spec.js file and Cypress will run the test for you. The result is shown below:

The test runs successfully! Let’s make sure the test results are displayed on the Applitools Test Manager website.

Let’s visit the Applitools Test Manager, and verify the results.

Great! Our test runs fine.

7. Running the Visual UI testing workflow

Run the following commands to push the new branch to Github:

git add .
git commit -m "Fix visual test script"
git push

Navigate to the GitHub repo, and create a new PR.

Notice how the CircleCI build pipeline has started:

ci/circleci:build Pending -- CircleCI is running your tests

Let’s wait for CircleCI to finish running the job and for Applitools to report the results of running the tests on this PR:

Both CircleCI and Applitools report success on running the single visual UI test!

Congratulations!

The ci/circleci:build job ran successfully
The scm/applitools job that compares the git branch to the baseline git branch ran successfully
The tests/applitools job that runs all the visual UI tests ran successfully

Only now you can go ahead and merge the PR into the source code.

Conclusion

This article is just the tip of the iceberg on how Applitools automated visual UI tests can seamlessly integrate with a CI engine to support and facilitate the DevOps workflow.

Now that you have a starting point, I will leave the floor to you. Explore, play and configure! Be brave and try more advanced workflows.

Cross-browser Testing With Cypress.io

Bilal Haidar — Wed, 11 Dec 2019 23:24:00 +0000

Cypress.io lets you test user interface design directly in the browser. But, Cypress only runs with Chrome. How do you do Cypress cross-browser testing?

Visual testing and cross-browser testing go hand-in-hand. With this tag team, your power to create superior, consistent, and pleasing Graphical User Interfaces (GUI) for your applications across a wide array of devices, and browsers, will seem like child’s play!

The huge wave of devices, operating systems, browsers, and screen resolutions flooding the market make Visual and cross-browser testing an essential constituent of any software development life-cycle. This ensures the stability of your UX designs across all of these devices.

Two leading tools in the field of software testing are Cypress.io and Applitools. Cypress.io is an open-source framework best known for its tools that help developers write integration and End-to-End (E2E) testing. Applitools, on the other hand, integrates with Cypress.io, giving testers, and developers, a chance to upgrade their test scripts into Visual ones.

While Cypress.io lacks support for cross-browser testing, Applitools supports it via their Applitools Ultrafast Grid.

You can combine the best of both worlds by using Cypress.io to write your integration and E2E tests and letting Applitools Ultrafast Grid handle cross-browser testing for you.

My goal, in this article, is to take you through a step-by-step guide that will include:

Writing a Cypress.io E2E test.
Running the E2E test locally using Google Chrome.
Converting the E2E test into a Visual test using Applitools Eyes Cypress SDK.
Running the Visual test across multiple browsers via the Applitools Ultrafast Grid.

Let’s start!

Welcome to Cypress.io

Cypress.io gives you a complete end-to-end testing experience by providing you with the right tools to set up your testing environment locally, write E2E tests, run the tests, and record them. It has been written from the ground-up without any dependency on other tools like Selenium.

To learn and use Cypress.io, I suggest watching the course Introduction to Cypress by Gil Tayar as he covers all the necessary information you need to get started using Cypress.io.

Show me an E2E test

I won’t keep you waiting any longer. Here’s a sample E2E test I wrote in Cypress.io:

View the code on Gist.

The sample E2E test script above starts by:

Navigating to the root page of the application.
Selecting two product items displayed on the page.
Verifying the two items selected are the ones displayed in the comparison section at the bottom of the page.

In the demo section below, I will delve into more detail on installing Cypress.io, setting it up for your projects, writing E2E tests, and much more. Stay tuned!

Oh, Cypress.io only works with Google Chrome!?!

Writing automated E2E tests with Cypress.io is fun and easy. Cypress.io has been gaining momentum in the software testing field due to its solid architecture, well-designed API and rich documentation just to name a few.

Cypress.io allows testers and developers to write E2E tests that run locally on the Google Chrome browser only. There has been debate, and requests from developers in general, to have this extend to other browsers as well. The Cypress.io team has listened, and are in the process of doing so.

You can follow the discussion on this GitHub Issue, Proposal: Support for Cross Browser Testing, for more information, and to stay up-to-date with the roadmap to support this. However, for now, you can only run your E2E tests locally with Google Chrome.

Is there a solution? Yes!

You can use Applitools’s Ultrafast Grid system to run Cypress.io E2E tests against multiple browsers and guarantee a cross-browser testing result.

Locally, you will be running your tests on Google Chrome while Applitools Ultrafast Grid will handle setting up the cross-browser environment to run your tests against multiple browsers.

Let’s see how it works!

Applitools to complement Cypress.io

Applitools offers a test framework to automate Visual testing of your application. It focuses on the visual aspects of your app and plays a major role in exposing the visual differences between baseline snapshots and both current and future snapshots.

Applitools integrates with dozens of testing frameworks, such as Cypress.io, Storybook, and Selenium. It provides SDKs for use in your projects to seamlessly communicate, and interact with Applitools.

The Applitools Ultrafast Grid runs your Visual tests quickly and supports a variety of configuration settings, such as multiple browsers, viewports, and devices. The Applitools Ultrafast Grid is split into two components: server-side and client-side. Here’s a diagram of how it works:

Applitools Ultrafast Grid – Setup Code

The Ultrafast Grid client is a Node.js library, used internally by both Applitools, Storybook and Cypress SDKs. Here’s more on the Applitools Ultrafast Grid client.

View the code on Gist.

That code snippet instantiates a new Grid client and returns an object. In this case, we are only interested in the openEyes() method returned.

Applitools Ultrafast Grid – Snapshot Capture

When called, the openEyes() method starts a new test run and sends the Grid backend server details about it. These details include the application name, batch or test name, the browsers used, and different variations of browser settings to be adapted while running the tests. This method returns a Promise to an object with the following functions: checkWindow() and close().

View the code on Gist.

To start collecting snapshots, the test issues a call to checkWindow() function. The Ultrafast Grid client handles this call and collects all the needed resources to store the snapshot locally.

At the end of the test, you issue a call to close() method to close and finish the test run.

Applitools Ultrafast Grid – Grid Tests

Launches a number of browsers with different viewports and device emulators.
Runs all of the DOM snapshots in parallel.
Collects image screenshots.
Sends all the screenshots to the Applitools AI server to do all the testing analysis and to return the results.

We will focus on the power of the Applitools Ultrafast Grid, to set up a testing environment in the cloud that allows running Visual tests against multiple browsers and resolutions.

Before I go further into writing and running Visual/E2E tests with Applitools Ultrafast Grid, let’s have a look at how Applitools integrates itself into Cypress.io.

Applitools/Cypress.io Integration

Applitools integrates with Cypress.io, and offers a general-purpose library that can be embedded within your project, whether it be the Angular, Vue, or React app.

The Applitools Eyes Cypress SDK is a simple plugin to Cypress.io. Once installed, it adds a few commands to the main cy object. Specifically, it adds three main methods: cy.eyesOpen to start the test, cy.eyesCheckWindow to take screenshots (for each test step) and cy.eyesClose to close the test. Here’s a diagram:

Now that you understand how the Applitools/Cypress.io integration works, let’s switch gears and start coding an E2E test.

Cross-browser testing step by step guide

Let’s get our hands dirty and build some E2E tests. In this section you will learn how to:

Install Cypress.
Write an E2E test.
Run the test.
Install Applitools Eyes Cypress SDK.
Write a Visual test.
Run that test.
Finally, do an analysis on the test results.

The E2E test I will be using for demonstration runs against one of the Applitools customers, https://www.salesforce.com. The test does the following:

Navigates the browser to SalesForce contact form located at: https://www.salesforce.com/uk/form/contact/contactme/
Fills the form fields with valid data.
Locates the Submit button and clicks it.
Finally, the test script verifies that the form has been successfully submitted by making sure the page displays a success message coming from the server.

Let’s start!

Create a new project folder

Start by creating a new folder by running these commands:

```

mkdir projects

cd projects

mkdir cross-browser-testing

cd cross-browser-testing

```

Now that you are inside the cross-browser-testing folder, run the following command to initialize NPM:

```

npm init -y

```

The command initializes your project with a package.json file. Now you can start installing NPM packages to use inside your project.

Install Cypress.io

Add Cypress package to the project by running this command:

```

npm install cypress --save-dev

```

The Cypress npm package adds a set of test files to help you write your own automated tests.

Open the Cypress Test Runner (Electron app) by issuing:

```

npx cypress open

```

This command opens the Cypress Electron app and also creates a cypress.json file and a cypress folder in your app’s root directory. The cypress folder is where we will be writing our tests.

To simplify our E2E tests, open the cypress.json file located at the root of your project and paste the following content:

```

{

"baseUrl": "https://www.salesforce.com/uk"

}

```

Now that Cypress.io is running properly, let’s add our first E2E test.

Write your first E2E test

Inside the cypress\integration folder, create the salesforce-contactus.spec.js file, and paste the following:

View the code on Gist.

The single E2E test focuses on testing the contact us form by filling its fields on the SalesForce website.

The spec file is self-explanatory. There’s just one thing to note here in relation to the selectors used inside cy.get() methods. The best practice is to avoid using IDs and CSS classes to select elements from the DOM. They will make your tests brittle because they’re subject to change. A better approach is to use data-* attributes, or actual component names.

In a controlled environment, I would stick to using data attributes. For now, we will keep on using the ID and Class CSS selectors for querying for DOM elements.

Run the test

The E2E test is ready, so let’s run it by issuing the following command:

```

npx cypress open

```

Cypress.io opens an instance of the Google Chrome browser to run your tests. Now, to run the same test against other browsers, we have to convert this E2E test into a Visual test and run it with Applitools Ultrafast Grid.

Let’s see how it works.

Install Applitools Eyes Cypress SDK

Let’s add the Applitools Cypress NPM package to the project.

Add the Applitools Eyes Cypress SDK package to the project by issuing the following command:

```

npm install @applitools/eyes-cypress --save-dev

```

Configure the Applitools Eyes Cypress SDK by issuing the command:

```

npx eyes-setup

```

Now that Applitools Eyes Cypress SDK is configured, we can start using it in our tests.

The Applitools Eyes Cypress SDK page has thorough documentation on how to use and customize the Applitools-Cypress integration, and how to configure the SDK to run tests against the Ultrafast Grid.

For this article, we will focus on how we can instruct the Visual Grid to run our test against multiple browsers and resolutions.

In general, there are three ways to specify test configuration:

Arguments to cy.eyesOpen().
Environment variables.
The applitools.config.js file.

You can read more about each method under the Advanced Configuration section.

I will make use of the first method, and supply a list of browsers and screen resolutions as arguments to the cy.eyesOpen() function.

Configure Applitools API Key

Before you can locally run Applitools Eyes Cypress SDK, make sure you get an Applitools API Key and store it on your machine as an environment variable.

To set the APPLITOOLS_API_KEY environment variable, you can use the export command on Mac, or set command on Windows as follows:

View the code on Gist.

Write your first Visual test

Now that you have the E2E test running, let’s convert this E2E test into a Visual test using Applitools Eyes Cypress SDK.

Revisit the cypress\integration\salesforce-contactus.spec.js file, and replace its content with:

View the code on Gist.

The code is self-documented.

Inside the beforeEach() test function, you start a new Visual test by calling the cy.eyesOpen({}) function that accepts a configuration object as input. You can specify the app name, batch name, browsers you want the Ultrafast Grid to use in order to run your tests, and many other configuration options available to you, on Applitools Eyes Cypress SDK page.

Close the current Applitools test by calling the cy.eyesClose() function inside the afterEach() test function.

Take a snapshot (You may repeat this step wherever you want to take a snapshot) by calling the cy.eyesCheckWindow() function. This function can take a snapshot name in case you want to distinguish snapshots from each other.

The test takes three snapshots at three different stages:

The blank SalesForce Contact Us page.
Once the SalesForce Contact Us form filled.
On the success confirmation page after submitting the Contact Us form.

Let’s run the test, and verify the results together.

Run the test

The Visual test is ready. Let’s run it by issuing the command:

“`

npx cypress open

“`

The Eyes Cypress SDK are all logged.

Let’s review the test results on the Applitools Test Manager.

Verify Test Results

Let’s visit the Applitools Test Manager, and verify the results.

On the left-side panel, there is a single test run named “fill the contact us form”. On the right-side panel, there are three test results- one for each “browser”. Remember, we’ve configured the Eyes SDK to run this Visual test across three different browsers/screen-resolutions:

View the code on Gist.

For each browser specified, the Grid spins a new instance of the browser and runs the test. This explains why we have three test results.

Applitools Ultrafast Grid takes the burden off your shoulders and establishes a cross-browser testing infrastructure in the cloud.

Each and every test result holds three snapshots. Let’s explore the snapshots for a single browser test result:

The first snapshot represents the form under test shown with empty fields.

The second snapshot represents the form under test shown with all fields filled.

The third snapshot represents the message shown to the user upon filling the “Contact us” form.

Using Applitools SDKs and Ultrafast Grid, we were able to run a single E2E test, written in Cypress.io, in a cross-browser/device testing environment!

Conclusion

Applitools, being a Visual Testing framework, fills the gap of E2E testing frameworks like Cypress.io by offering cross-browser testing. Its UltrafastGrid can set up a cross-browser testing environment in the cloud and can run your tests there.

How Cross-Browser Testing Is Evolving

Bilal Haidar — Tue, 29 Oct 2019 01:26:39 +0000

Today I’m going to write about how cross-browser testing is evolving. And I’ll do that with Cypress. If you know Cypress, you know it only runs on Google Chrome. So, how can anyone leverage Cypress in cross-browser testing?

Want to know more? Read on.

Functional testing has been around long before the introduction of Visual and UI Testing. Functional tests, also thought of as end-to-end (e2e tests), sit at the top of the software testing pyramid as shown in Figure 1.

The pyramid applies to both manual and automated testing.

The introduction of Visual testing has seen its steady rise to the top tier, side by side with functional testing.

There is a fine line between writing E2E tests compared to writing Visual tests. But don’t worry, we will cover this topic in more depth. However, the desired end result is the main difference between the two.

Testing What Your User Sees

Visually testing your application across multiple browsers and devices is a building block when referring to visual testing. Maybe the most important, and ultimate goal, is the appearance of your application across multiple browsers, devices and screen resolutions.

Visual testing can be easily integrated into your functional testing with little effort thanks to tools like Applitools. Although writing functional and visual testing may be similar, each has a purpose with a different goal in mind.

Having both functional and visual testing at the top of the software testing pyramid, the question has to be asked:

Should functional testing be concerned with cross-browser testing?

Should visual testing be concerned with cross-browser testing?

There is no direct or specific answer. The topic has been debated with many varying opinions.

Here, I will give my analysis in the hope of shedding light on this topic. You’ll find out ways to go about functional and visual testing in software development.

What is Functional Testing?

Functional testing is a type of black-box software testing in which software is tested against functional requirements/specifications with the goal of finding a bug/defect(s) if any.

Functional testing consists of 6 steps:

The identification of functions that the software is expected to perform
The creation of input dataset based on the function’s specifications
The determination of output based on the function’s specifications
The execution of the test case
The comparison of actual and expected output
To check whether the application works as per the customer’s need

Functional testing is tightly coupled with the software functional requirements. At the end of the day, this is what you are testing. You are making sure the client’s brief has been fulfilled in the application.

Take for instance the following case:

A login page in your application
The user has to enter their details and hit submit to login
Successful login should redirect the user to the homepage of the application

You take the above functional requirement, and dissect it to find:

The input of this operation is the user details
The output of this operation is that the user is redirected to the homepage upon a successful login

Whether you are doing manual or automated testing, you follow the same steps in performing functional testing.

One of the most popular frameworks to date, for functional or E2E testing, is Cypress.io.

Below is a sample Cypress.io script that automates the functional/E2E test:

View the code on Gist.

The script is straightforward and easy to follow:

Opens the login page of the application
Fills in the username and password
Locates and submits the form
Waits for the page to load, and checks if the “homepage” text is found on the page

If the login is successful, the application passes the requirements.

Now that you have an idea of what Functional or E2E testing is, let’s look at Visual testing.

What is Visual Testing?

Visual testing addresses the behavior of the Graphical User Interface of your application, making it aesthetically pleasing to the eye. That is, the positioning of elements on the page, the colors, and many visual appearance aspects. This would include: running the same visual test across multiple browsers, devices, screen resolutions, operating systems, etc.

Visual testing is regression testing that follows certain steps to validate that the screens, or pages, have not changed unexpectedly from one test run to another, or due to codebase changes introduced since the last test run.

With the E2E test we just looked at, adding visual tests is a no brainer. In general, Visual tests are built on top of E2E tests. Yes, you can take an existing functional test, integrate a visual testing library into the mix and you got visual testing up and running.

Example: Visual Testing with Cypress and Applitools

For instance, you can start writing visual tests by using Applitools SDKs that integrates seamlessly with Cypress.io E2E tests.

Let’s take the same E2E test, and add a few steps to insert a visual test:

Taking the same Cypress.io test script, you can add Applitools SDK calls to take snapshots at different stages of running this script:

View the code on Gist.

Do you notice how visual test scripts enhance E2E tests, so they complement each other?

That’s not the whole story! Now the state of the DOM captured along with the visual snapshots get sent to the Applitools Ultrafast Grid in the cloud. Based on how you setup your test, the Ultrafast Grid would configure and start a new testing environment – different browsers, device emulators, screen resolutions. It would then run your DOM snapshots, generate images, and finally, sends them to the Applitools AI server.

The AI server will do all the comparisons for all the snapshots by comparing a baseline snapshot (if it exists) to the current image snapshot. Depending on the comparison results, you may keep the baseline snapshot (if it existed in the first place) or make this the baseline snapshot to compare to the next time a visual test is run or decide this latest snapshot will be the new baseline to compare against.

This is a general overview of how visual testing is done when using Applitools, and has the capacity to test many times over.

The takeaway is that visual tests make use of E2E tests. By employing visual testing tools like Applitools, the return on writing E2E tests is doubled. You are writing your E2E tests once. As a result, you are now performing functional and visual testing to make sure your application passes both the functional and visual appearance tests.

Cross-Browser Automation Testing

What is it and why do we need it?

Cross-browser automation testing is the act of running your automated tests across multiple, and different browsers, belonging to different platforms, and devices.

Generally, in web development, an essential part of our job is to ensure that the JavaScript and CSS we write are cross-browser. That means that the application runs smoothly across all platforms.

You can relate this concept to automation testing. When you run your automated test scripts across multiple browsers, you are safeguarding your application to work properly across the board.

The above is not 100% true when it comes to functional or E2E tests. However, this is always true when discussing cross-browser visual testing.

Let’s delve further, and compare which method is more effective when testing our applications by looking at cross-browser functional, or cross-browser visual, testing.

Cross-Browser Functional Tests

You’ve learnt that functional, or E2E, tests deal with validating the logic of the application, and making sure the function satisfies the specifications given to you by your client.

A functional test follows a step by step path in your application, allowing it the ability to validate its primary function.

An example of this might be in an e-commerce application validating a credit card number being entered by the user at the checkout. In the case of an SPA (Single Page App) your JavaScript code is the validator.

Another example, mentioned earlier, where a user enters their details on a Login form, and is redirected to the homepage, is the responsibility of the JavaScript code if it’s in the form of a SPA.

Yet another example of an E2E test is to make sure the Logout button is visible on the top navigation bar once a user is logged in. This feature involves writing a function to check the LocalStorage and show the Logout button. Once again, this E2E is testing the login process that is controlled by writing JavaScript code if it is a SPA.

The point I am making is this: if you follow the strictest procedure when writing your JavaScript code, and use native JavaScript APIs that are certified cross-browser when selecting third party libraries that are cross-browser compatible, you are making sure that your E2E tests (once they run and pass against one browser) will run against other browsers too. There are exceptions to the rule, but only a small percentage of your E2E tests.

These exceptions might be a result of using third party libraries, or native JavaScript APIs, that are known to have issues on some browsers. You might opt to run your E2E tests on another browser to establish how the application fails, and try to provide a solution.

Establishing a cross-browser lab to run your tests is expensive. This involves setting up multiple computers, each with different operating systems, multiple devices, phones and tablets. It’s just not feasible. Add up costs, time, and the never-ending advances in technology, and you will agree, it’s a nightmare.

When using Applitools Ultrafast Grid, you can forget cross-browser functional testing because eventually your E2E tests will be converted to Visual tests. Ultrafast Grid will do the job of running your E2E tests across multiple browsers. More on this shortly.

Cross-Browser Visual Testing

Cross-browser visual testing is a must. You are testing the visual appearance of your application rather than the function of it.

Visual testing is about testing the placement of elements on the page, the coloring of elements, the relation between elements when the screen size changes. For instance, do elements overlap each other? Do they obscure each other? How responsive is your page for different screen resolutions? How responsive is your page on smaller devices? Bigger devices?

All these questions cannot be answered by doing cross-browser visual testing. Despite the fact you might be using cross-browser CSS frameworks like Bootstrap, the probability of using CSS selectors and styles that are incompatible is highly likely. There’s a ton of CSS stuff that’s available in one browser and not on another. You are more prone to error by using non-cross-browser CSS styles than non-cross-browser JavaScript code.

Therefore, it makes perfect sense to use Applitools Ultrafast Grid to solve this issue. You will see how inexpensive and reliable it truly is to perform cross-browser visual testing.

Why Applitools Ultrafast Grid?

If you are new to Ultrafast Grid, read my article titled How I ran 100 UI tests in just 20 seconds .

In brief, Ultrafast Grid is composed of two components:

Client-side component, representing SDKS, and library, to be used when writing visual tests, to capture DOM snapshots, and send these snapshots to the server-side component living in the cloud.
Server-side component is usually hosted in the cloud, and receives DOM snapshots from its counter client-side component. Based on a set of configuration settings, Ultrafast Grid will establish a cross-browser testing environment, run all of your DOM snapshots there, generate image snapshots, and, finally, send those snapshots to the Applitools AI Engine that takes over the mission of doing image comparison and analysis.

When you employ Ultrafast Grid in your visual testing, you usually configure the settings of your test runs.

An example of such a configuration file is this:

View the code on Gist.

This configuration file instructs Ultrafast Grid on what browsers and devices are needed for your tests. Ultrafast Grid runs all your visual tests in all the browsers, operating systems, and screen resolutions you specify. You see how inexpensive it is to use Ultrafast Grid for cross-browser testing?

Not only this, with Ultrafast Grid, your visual tests run faster. You are no longer sending the server image snapshots you have captured on the client-side, but rather DOM snapshots containing the required resources from HTML, CSS and JavaScript that are enough to re-generate on the server (by the Ultrafast Grid) in order to generate the real image snapshots.

By utilizing the Ultrafast Grid, you get a bonus! As mentioned, when writing visual tests in Applitools, you are taking your functional, or E2E tests, adding to them a few calls to hook into the Applitools SDKs, and you are ready to start visual testing. So eventually, and indirectly, your E2E tests are being run on multiple browsers, and you get a chance to run your functional tests on multiple browsers inexpensively.

Conclusion

Cross-browser testing is not a “Yes” or “No” thing. Many factors play into whether or not you should do it, especially when you are dealing with functional, or E2E testing.

However, it becomes mandatory when you are doing visual testing.

Establishing an efficient, reliable, and accurate testing environment has been made very simple if you opt to use tools like Applitools Ultrafast Grid.

For More Information

Read more about “What Is Visual Testing?”
Look at Gil Tayar’s class to learn Cypress on Test Automation University.
Enroll in Test Automation University (it’s free).
Read my step-by-step tutorial for using React and Cypress with Applitools.
Read my step-by-step tutorial for using Vue and Cypress with Applitools.
Again, a link to my blog post about using Ultrafast Grid to run 100 UI tests in 20 seconds.
View the Applitools tutorials.
Sign up for an Applitools account (it’s a free account).
Request an Applitools demo.

The post How Cross-Browser Testing Is Evolving appeared first on Automated Visual Testing | Applitools.

Vue.js Development with Storybook and Applitools

Bilal Haidar — Tue, 17 Sep 2019 01:06:03 +0000

At one point in time, the server-side code was responsible for generating and serving every page in our application. Nowadays, Single Page Apps (SPA) are taking a bigger slice of the pie, as more clients are leaning towards building their applications as an SPA.

Developers spend most of their time debugging a SPA inside the browser, by debugging the JavaScript files, CSS styles and the DOM, hoping to find the root cause of any visual bug appearing in their application.

Applitools realizes the complexity of visually debugging an application and introduced the Root Cause Analysis (RCA) module on their Test Manager Dashboard to help quickly identify the root cause behind visual UI changes.

In this article, I will demonstrate the Applitools RCA feature by walking you through a complete step by step guide to visually test a Vue.js app with Storybook and Applitools.

If you’re not already familiar, Applitools is an automated visual regression testing framework. It focuses on the visual aspects of your app and plays a major role in exposing the visual differences between baseline snapshots and both current and future snapshots.

Applitools integrates with dozens of development and testing frameworks, such as Cypress.io, Storybook and Selenium. It provides SDKs for use in your projects to seamlessly communicate and interact with Applitools, in order to send both baseline and test screenshots.

So, get started with visual UI testing. Your visual testing project will depend on React, Storybook and Applitools. You will also need the node.js tools. So, firstly, make sure you know the following software tools and install on your machine:

Before we immerse ourselves in writing code, let’s look at how Applitools and Storybook work together.

How Applitools Works With Storybook

Whether you use Vue.js, React or Angular, Storybook doesn’t offer any extension points for third-party frameworks, such as Applitools, to integrate with. The Applitools team worked around this limitation by providing the Applitools Eyes Storybook SDK.

Once installed, this SDK collects all stories in the application, runs them all via the Storybook engine, generates DOM snapshots for each story and uploads these snapshots to the Applitools Eyes server. Here’s a visual of this flow:

The Applitools server runs the tests generates image snapshots. Applitools compares each snapshot to a baseline snapshot (if one exists) and displays the results on the Test Manager Dashboard.

This kind of integration provides a smooth experience in terms of writing Storybook stories. Nothing changes in the way you write your Vue.js apps.

Automated Root Cause Analysis

The concept of Root Cause Analysis stems from the world of management, where it’s defined as a method of problem solving, used for identifying the root causes of faults or problems.

So far, Applitools has been focusing on visual testing by comparing the baseline and the current test run snapshot through their AI engine. It finds the visual differences in order to map them graphically. Previously, there was no way to search your codebase for a reason behind any visual testing bug. As you may know, searching through code is nightmarish, time-consuming and tedious!

This is where Applitools steps in with RCA, showing you the exact DOM and CSS changes between the baseline snapshot and the current test run snapshot.

Why is this so important for both developers and QA testers? Let’s take a look.

Step By Step Guide

For this demonstration of visual testing a Vue.js application with Storybook and Applitools, I’ve chosen the Bootstrap Vue open-source library hosted on GitHub under this repo. This library, with over 40 plugins and more than 75 custom components, provides one of the most comprehensive implementations of Bootstrap v4 components and grid system for Vue.js. With extensive and automated WAI-ARIA accessibility markup.

The plan is to start by installing Storybook onto the Bootstrap Vue project. Write and run a few Storybook stories, install the Applitools Eyes SDK for Storybook and finally demonstrate the RCA feature.

Now, let’s get on with it and start coding …

Step 1: Clone the Bootstrap Vue GitHub repo locally on your machine by running this command:

git clone git@github.com:bootstrap-vue/bootstrap-vue.git

Step 2: Install all NPM package dependencies by running this command:

yarn

Step 3: Install the Storybook NPM package as a dev-dependency by issuing this command:

yarn add @storybook/vue -D

Step 4: Install all peer dependencies required by Storybook to function properly by issuing this command:

yarn add vue-loader babel-loader babel-preset-vue -D

Step 5: Install css-related NPM packages in order to load CSS styles inside your Storybook stories by issuing this command:

yarn add css-loader style-loader -D

Step 6: Add the following NPM script to your package.json file in order to start the Storybook:

{

  "scripts": {

    "storybook": "start-storybook"

  }

}

Step 7: Create the Storybook config.js file inside a .storybook folder located at the root of the solution folder as follows:

import { configure } from '@storybook/vue';

function loadStories() {

  const req = require.context('../stories', true, /\.stories\.js$/);

  req.keys().forEach(filename => req(filename));

}

configure(loadStories, module);

Step 8: Create a scss-loader.scss file inside the .storybook folder to include all the styles required by Bootstrap Vue library components to load properly inside our Storybook stories:

@import ‘../node_modules/bootstrap/scss/bootstrap.scss’

@import ‘../src/index.scss’

Then import this file inside the config.js file as follows:

import { configure } from '@storybook/vue';

import ‘!style-loader!css-loader!sass-loader!./scss-loader.scss’

function loadStories() {

  const req = require.context('../stories', true, /\.stories\.js$/);

  req.keys().forEach(filename => req(filename));

}

Step 9: Create the navbar.stories.js file inside the stories folder, located at the root of the solution folder, to hold our Storybook stories for the NavBar component:

import { storiesOf } from '@storybook/vue'

const navbarStories = storiesOf('BNavBar', module)

navbarStories.add('Full Navbar', () => ({

  template: `

  

    NavBar

    

    

      

        Link

        Disabled

      

      

        

          

          Search

        

        

          EN

          ES

          RU

          FA

        

        

          

          Profile

          Sign Out

        

      

    

  

`

}))

The story groups all NavBar stories under the name BNavBar and assigns the first story the name of Full Navbar.

There are two ways to tell Storybook stories about your Vue.js components. Either define all used components in a components object property on the story itself or define the components globally inside .storybook/config.js file so that they are available to all running stories in your solution.

Let’s follow the latter option and define all components needed by NavBar story inside .storybook/config.js file using the Vue.component() helper method:

import { configure } from '@storybook/vue'

import Vue from 'vue'

import '!style-loader!css-loader!sass-loader!./scss-loader.scss'

import {

  BNavbar,

  BNavbarNav,

  BNavbarBrand,

  BNavbarToggle,

  BButton,

  BCollapse,

  BNavItem,

  BFormInput,

  BNavForm,

  BNavItemDropdown,

  BDropdownItem

} from '../src'

Vue.component('b-navbar', BNavbar)

Vue.component('b-navbar-nav', BNavbarNav)

Vue.component('b-navbar-brand', BNavbarBrand)

Vue.component('b-navbar-toggle', BNavbarToggle)

Vue.component('b-button', BButton)

Vue.component('b-collapse', BCollapse)

Vue.component('b-nav-item', BNavItem)

Vue.component('b-form-input', BFormInput)

Vue.component('b-nav-form', BNavForm)

Vue.component('b-nav-item-dropdown', BNavItemDropdown)

Vue.component('b-dropdown-item', BDropdownItem)

function loadStories() {

  const req = require.context('../stories', true, /\.stories\.js$/)

  req.keys().forEach(filename => req(filename))

}

configure(loadStories, module)

Step 10: Now the story is ready, let’s run it with Storybook using this command:

yarn storybook

The Storybook engine runs successfully and renders our Navbar story.

Now let’s install the Applitools Eyes SDK for Storybook.

Step 11: Install the Applitools Eyes SDK for Storybook NPM package as a dev-dependency by issuing this command:

yarn add @applitools/eyes-storybook -D

In order to authenticate via the Applitools server, you need to supply the Eyes-Storybook SDK with the API key you got from Applitools. Read more about how to obtain the API key here.

To do this, set the environment variable APPLITOOLS_API_KEY to the API key before running your tests. For example, on Linux/Mac:

export APPLITOOLS_API_KEY=

And on Windows:

set APPLITOOLS_API_KEY=

Step 12: Let’s run the first cycle of the visual UI testing by issuing this command:

npx eyes-storybook

Once the tests are done running, let’s check back with the Applitools Test Manager Dashboard and verify the results.

The left side of the Dashboard lists all batch test runs that you have previously performed. Currently, there is a single test run with a status of Passed. This is the first test run and so Applitools engine automatically passes this test and all snapshots thereafter as baseline snapshots.

Click on the single batch test to view the snapshots available. For each and every Storybook story, a snapshot is generated by Applitools.

Expand the BNavBar: Full Navbar test run and click the snapshot to view it in a bigger scale:

As you can tell, the test run snapshot is simply an image snapshot for the story that runs inside the Storybook playground application.

At this moment of time, Applitools stores all the generated snapshots as baseline snapshots. The future test runs would generate a new set of snapshots and Applitools would compare them to the baseline ones and report the results accordingly.

You have noticed that Applitools Eyes SDK for Storybook is prepackaged with some default settings allowing you to do visual testing for your stories without having to specify any test configuration settings. However, that doesn’t mean that Applitools doesn’t offer you the means to customize your tests by specifying test configuration settings either through environment variables or storing your settings inside the applitools.config.js configuration file.

Let’s switch gears and explore a unique gem of Applitools that is the Regression UI testing. The traditional regression testing verifies recent source code changes or additions and that the functionality of the software has not changed or been affected in any way.

Applitools allows you to perform visual UI Regression testing. You can run multiple cycles of test runs and all visual changes collected from one to another, are detected by Applitools servers and logged into the Dashboard.

Step 13: Let’s introduce a visual change in the Navbar component by including the following CSS selector inside .storybook/scss-loader.scss file and run our tests again.

@import "../node_modules/bootstrap/scss/bootstrap.scss";

@import "../src/index.scss";

a {

  background-color: yellow;

}

The change causes all the hyperlinks on the Navbar component to render with a background color of yellow.

Step 14: Let’s run the second cycle of the visual UI testing by issuing this command:

npx eyes-storybook

Notice the logs on the terminal windows after running the command:

The test failed and that’s exactly what we expect. Applitools detected the change we introduced to the background color of the hyperlinks and reported it.

Let’s switch back to the Applitools Test Manager Dashboard to check the results.

The new test run is marked as Unresolved. Expand on the Navbar test run to view the visual differences highlighted to you by Applitools:

Make sure both snapshots are selected so that you can easily view the differences. If not, locate the following menu and choose the option Both:

The snapshot above highlights all the hyperlinks, indicating that those hyperlinks have changed.

Step 15: Click on the RCA tool icon (highlighted in the red rectangle) to get a full analysis of the source of this visual bug.

Next, you select the visual difference you want RCA to assess.

I’ve selected to click on the Lang hyperlink on the Navbar. Notice how the RCA tool detects a css style change on the tag represented by the Yellow background.

The RCA tool runs and shows the root cause of this visual difference, which is a change in the CSS Rules.

Step 16: The RCA tool can also detect Attribute changes. Let’s change the tag of the Search box to have a size of lg as follows and run the test once more.

The RCA tool detects a change in the attributes of the Search box.

It also shows the change of CSS rules as follows:

Step 17: Another category RCA detects is the DOM changes category. Let’s wrap the Search button text with a paragraph tag as follows and run the test again:

Search

The RCA tool detects a change in the text inside the Search button. Instead of rendering the text as is, the new test snapshot renders the text inside a

tag.

Any test run with status Unresolved requires someone’s attention and manual intervention to accept or reject the change. In this case, you may accept the new change(s) and thus update the baseline snapshots.

On the other hand, if the change detected is undesirable, then this implies a bug in the source code and you can reject this new test run to keep the original baseline snapshot without any changes.

In Conclusion

Applitools RCA is both powerful and convenient. It can pinpoint the exact DOM/CSS change. If you did not intend the change, you can identify the code the responsible code quickly and make appropriate modifications to pass your tests.

RCA supports a set of DOM/CSS categories of change, including:

Textual changes
CSS Property changes
Attributes changes
Bounding Box changes
Tag changes (for example, when you replace a button with a hyperlink)

Happy Testing!

Video Demonstration

More about Applitools

The post Vue.js Development with Storybook and Applitools appeared first on Automated Visual Testing | Applitools.

JavaScript Visual UI Testing with Selenium [step by step]

Bilal Haidar — Fri, 06 Sep 2019 20:27:27 +0000

JavaScript? It’s a standard. Selenium? It’s this great WebDriver. So, why do I get visual errors?

In this article, I will go back to basics and share a step by step tutorial on how to UI test for native JavaScript applications using Applitools and its powerful integration with the Selenium WebDriver library.

Applitools is an automated visual regression testing framework. It focuses on the visual aspects of your app and plays a significant role in exposing the visual differences between baseline snapshots and both current and future snapshots.

Applitools integrates with dozens of testing frameworks, such as Cypress.io, Storybook, and Selenium. It provides SDKs for use in your projects to seamlessly communicate and interact with Applitools.

Terminology

Throughout the article, I will use some terminology repeatedly. You need to know these terms to get the most from this article.

WebDriver

“WebDriver is a remote control interface that enables introspection and control of user agents. It provides a platform and language-neutral wire protocol as a way for out-of-process programs to remotely instruct the behavior of web browsers.” WebDriver W3C Recommendation 05 June 2018.

WebDriver aims at writing a standard set of instructions to query for DOM elements and to control the behavior of browsers in a cross-browser and cross-platform fashion.

Chrome, FireFox, IE, Edge, and many other browsers provide their own implementation for the WebDriver Standard. Hence, they implement the same standard API and provide browser-specific functionality.

Selenium WebDriver

“Selenium is a browser automation library. Most often used for testing web-applications, Selenium may be used for any task that requires automating interaction with the browser.” Selenium WebDriver NPM

Selenium WebDriver is a layer on top of the WebDriver standard that provides seamless programmatic access to any browser that supports a WebDriver implementation. Once you have ChromeDriver installed on your machine in its correct path, a Selenium script will detect the ChromeDriver automatically. Selenium can then send all the instructions to the ChromeDriver to control the Chrome browser. Similarly, if you wish to use FireFox and install the FireFox Driver, Selenium would automatically interact with that driver to control the FireFox browser. Rather than requiring developers and testers to maintain separate WebDriver implementations, Selenium Webdriver provides a standard API for running test scripts multiple browsers and platforms.

Applitools Eyes SDK for Selenium and JavaScript

Applitools Eyes SDK is a thin layer on top of Selenium WebDriver. The Eyes SDK extends Selenium WebDriver and lets developers and testers the tools to incorporate visual and UI testing into their applications.

To write a visual e2e (end-to-end) test using the Applitools Eyes SDK, the script starts by initializing a Selenium driver object that is passed to the eyes.open() method as a first parameter. The SDK wraps the driver with easy to use methods. This allows the developer to start a test, capture a snapshot and finally close the test.

We’ll cover Applitools Eyes SDK in the step by step guide below.

Source code

In this article, I will demonstrate the Applitools Eyes SDK by building a visual e2e test script to test out the Wikipedia website. The script starts by navigating the browser to the Wikipedia website. It then locates and clicks on the English version of Wikipedia, searches the Wikipedia for the term “Software Testing” and finally verifies the results. The source code for this article is available on this Github repo. Wikipedia Testing with Applitools and Selenium.

Prerequisites

First, ensure that these are properly installed on your machine:

Before we start coding, let’s ensure that you’re ready with ChromeDriver and Applitools.

Prerequisite Steps 1-4: ChromeDriver / Chrome Browser

Step 1: Install Google Chrome browser from https://www.google.com/chrome/

Step 2: Install a Chromedriver that’s appropriate for your operating system and your Chrome browser’s version from:

http://chromedriver.chromium.org/downloads.

For example, if your Chrome browser is v67-v69, download ChromeDriver 2.41.

Step 3: Unzip ChromeDriver to any folder of your choice.

Step 4: Add ChromeDriver’s path to your system’s PATH so Selenium can find it.

On a Mac:

Open ~/.bash_profile file (Create one if it’s missing).
Add export PATH=”:$PATH”
Run source ~/.bash_profile.

For example, if the ChromeDriver is in /Users/apps/chromedriver, then it would be export PATH=”/Users/apps/:$PATH

On a Windows:

Navigate to Computer (Right click) | Properties | Advanced System Settings | Advanced (Tab) | Environment variables | System Variables.
Select the line with Path key.
Click on Edit.
Add a semicolon ; to the end of the values and add the path where you have unzipped ChromeDriver.

For example, if the ChromeDriver is in c:\selenium\chromedriver, then append ;c:\selenium to the end of the Path. Finally, save it.

If everything went fine, and if you typed chromedriver in the command prompt, you should see something like this:

Prerequisite Step 5: Configure Applitools API KEY

Before you can locally run Applitools Eyes SDK, make sure you get an Applitool API Key and store it on your machine as an environment variable.

To set the APPLITOOLS_API_KEY environment variable you can use the export command on Mac or set command on Windows as follows:

View the code on Gist.

Demo Steps 1-3: Clone, Install Selenium and Applitools SDK

The section below will guide you step by step in writing your first Applitools Selenium e2e test.

Step 1: Clone the application repo.

git clone git@github.com:bhaidar/wikipedia-testing-applitools-selenium.git

Step 2: Install the Selenium WebDriver package as a dev dependency by issuing the following command:

npm install selenium-webdriver --save-dev

Step 3: Install the Applitools Eyes SDK for Selenium as a dev dependency by issuing the following command:

npm install @applitools/eyes-selenium --save-dev

Demo Steps 4-7: Prepare JavaScript for E2E Test With Selenium

Step 4: Let’s start creating our first e2e test. Create a new file wikipedia.spec.js at the root of the solution folder and paste the following two lines at the very beginning:

View the code on Gist.

The code snippet starts by importing a few objects from the Selenium WebDriver NPM package. The second line imports the main Eyes object from the Eyes SDK for Selenium.

Step 5: Build a new driver object inside an IIFE (Immediately Invoked Function Expression) so that our code is isolated from the Global environment in JavaScript.

View the code on Gist.

The code snippet makes use of the Selenium WebDriver API to build a new Chrome browser object.

Step 6: Initialize an instance of the Eyes object as follows:

View the code on Gist.

The code simply creates a new object instance of the Eyes class.

The code snippet then sets the Applitools API Key by making use of the APPLITOOLS_API_KEY environment variable.

Finally, the code configures the Eyes object instance to show logs while running. I enable logging intentionally while writing my e2e tests. As a result, I get the chance to see all the logs generated by Eyes SDK that could help me trace any bug in my code.

Step 7: Let’s start a new Eyes SDK test by calling the eyes.open() method as follows:

View the code on Gist.

The code snippet above defines a try/catch block so that we guarantee a safe closure for any opened test.

To start using the Eyes SDK, you call the eyes.open() method passing to it the WebDriver object defined above together with test configuration settings. Calling this method initiates communication with the Applitools backend servers to start processing your tests.

Remember, the Eyes SDK for Selenium is a thin wrapper on top of the Selenium WebDriver; hence, the need to pass the driver object to the Eyes SDK.

The App Name, Test Name, and Viewport settings are the essential configuration settings that you need to pass to the eyes.open() method.

Towards the bottom of the try block, it is essential to close the test safely. The finally block closes the browser instance opened by the Selenium WebDriver and makes sure the Eyes test is completely closed by aborting it in case it is still pending.

Now with that out of the way, let’s write our test!

Demo Steps 8-14: Create Applitools Checks In the Selenium JavaScript Test Script

Step 8: Navigate to the Wikipedia Website by adding this line of code into your e2e test script:

View the code on Gist.

Step 9: Take a visual checkpoint using the Eyes SDK by adding this line of code into your e2e test script:

View the code on Gist.

The eyes.check() method accepts a simple string that labels the snapshot taken.

Step 10: Locate the “English” hyperlink on the Wikipedia Home page and click it with this line of code:

View the code on Gist.

The browser navigates to the English version of Wikipedia.

Step 11: Verify that the page the browser navigated to, has a header with text of “Welcome to Wikipedia”:

View the code on Gist.

Step 12: Take a visual checkpoint using the Eyes SDK by adding this line of code into your e2e test script:

View the code on Gist.

Step 13: Locate the search box, type the text “Software Testing” and hit Enter. The code below demonstrates the three steps:

View the code on Gist.

Step 14: Wait until the Wikipedia Software Testing page loads and then take a new snapshot:

View the code on Gist.

Every page in Wikipedia has a title element with ID of “firstHeading”.

Finally, we close the current test and quit the browser.

Demo Steps 15-16: Verify Test Results for Selenium and Javascript in Applitools

Step 15: Run the e2e test script by issuing this command:

node wikipedia.spec.js

As you can see this is a simple Node.js script that starts executing the instructions from top to bottom.

As a result, you should see the following output once the script runs.

Step 16: With that done, we’ll visit the Applitools Dashboard and verify the test run results.

Navigate to https://www.applitools.com and Sign In to access your account.

The Dashboard lists all test runs on the left-side of the page. By clicking a test run, you can then check all the snapshots that were taken during the test run.

In our case, there are three snapshots representing the Wikipedia Home page, Wikipedia English page and Wikipedia Software Testing page.

The test has a status of Passed. The snapshots are stored as baseline snapshots and everything seems to be working fine.

Let’s now introduce a visual change in the test run by running our e2e test to demonstrate how visual regression testing works with Applitools.

Demo Steps 17-18: Make Changes And Validate Differences

Step 17: Let’s change the search term from “Software Testing” to “Software Testability” and run the e2e test again by issuing the following command:

node wikipedia.spec.js

Running the above command results in the following error:

That’s great news! The Applitools Eyes SDK was able to detect visual differences between the two test runs and that’s exactly what we want.

Step 18: Let’s switch back to the Applitools Dashboard and check the new test run results there.

Notice how Applitools was able to detect some visual changes in the third snapshot taken.

Let’s expand on the third snapshot to see the visual differences detected:

The visual differences are clear between the snapshots. One snapshot displays the Software Testing page and the other displays the Software Testability page.

The power of Applitools lies in the visual regression testing capabilities that make it not only a visual testing framework, but an obvious choice to run all regression tests in your application. Applitools will ensure there are no intentional changes across the life cycle of the software under development.

Here’s the source code repo for completed e2e test application.

Conclusion

You now have some insight on how Applitools speeds up your visual testing experience when you’re developing apps with native JavaScript/Node.js and Selenium WebDriver.

My upcoming articles will explore Applitools further, unveil more features and allow you to carry on with E2E and Visual UI Testing easily.

Happy Testing!

Want To Know More About Applitools?

You can request a demo.
You can sign up for a free account.
You can check out their tutorials.

The post JavaScript Visual UI Testing with Selenium [step by step] appeared first on Automated Visual Testing | Applitools.

Cypress and Angular Testing with Applitools

Bilal Haidar — Wed, 31 Jul 2019 16:16:49 +0000

Cypress and Angular make a great combination. Angular is a useful and broadly-adopted front-end framework, focused on single-page applications. Cypress makes it easy to set up, write, run, and debug tests.

Applitools is an automated visual regression testing framework. It focuses on the visual aspects of your app and plays a major role in exposing the visual differences between baseline snapshots and both current and future snapshots.

Applitools Eyes integrates with dozens of testing frameworks, such as Cypress.io, Storybook, and Selenium. Applitools provides SDKs for use in your projects to seamlessly communicate and interact with Applitools Eyes.

Getting Started with Cypress, Angular, and Applitools

With all the attention on Angular visual UI testing, I thought it would be appropriate to share step by step tutorial on this subject. Specifically, Angular UI testing using Applitools and Cypress.io.

First, ensure that these are properly installed on your machine:

Before we delve into writing code, let’s take a behind-the-scenes look at how Applitools and Cypress.io integrate together.

Applitools / Cypress.io Integration

Applitools integrates with Cypress.io and offers a general-purpose library to embed within your project, whether it be the Angular, Vue or React app.

The Cypress library, Applitools Eyes Cypress SDK, adds a few commands to the main cy object. Specifically, it adds three main methods: cy.eyesOpen to start the test, cy.eyesCheckWindow to take screenshots for each test step and cy.eyesClose to close the test. Here’s a diagram:

Now that you understand how the Applitools/Cypress.io integration works, let’s start coding a few visual tests for an Angular app using Applitools and Cypress.io.

Source code

For this article, I’ve chosen to write a few E2E Cypress.io tests against the Angular Pizza Creator app written by Todd Motto which is available on Github. The Pizza Creator app builds a pizza ordering form that basically allows the client to customize the pizza ingredients before ordering it. I’ve cloned the Github repo, created a new Angular CLI app and migrated all the source code there. The new Github repo can be found here.

Now, I will take you on a series of steps to install Applitools and Cypress.io on an existing Angular app, add a Cypress.io test and finally verify the results on the Applitools Test Manager.

Steps 1-3: Prepare The Demo App

Step 1: Clone the repository locally by issuing the following git command:

git clone git@github.com:bhaidar/pizza-creator.git

Step 2: Install all the npm dependencies by issuing the following command:

yarn install

Step 3: Run the app:

yarn run start

And you should see something like this:

There you go. It works!

Steps 4-7: Install And Configure Cypress

Step 4: Add Cypress package to the project:

yarn add cypress -D

The Cypress npm package adds a set of test files to help you write your own automated tests.

Step 5: Replace the content of the e2e script inside the package.json file with:

“e2e”: “cypress open”

Step 6: Open the Cypress Electron app by issuing the following command:

yarn run e2e

This command opens the Cypress Electron app and also creates a cypress.json file and a cypress folder in your app’s root directory. The cypress folder is where we will be writing our tests.

Step 7: Inside the cypress.json file in the root of the application folder, add:

{

“baseUrl”: “http://localhost:4200”

}

Now that Cypress is running properly, let’s add our first E2E test.

Steps 8-9: Install and Run End To End Test

Step 8: Inside the cypress\integration folder, create the pizza.spec.js file and paste:

View the code on Gist.

The single E2E test focuses on testing the function of the pizza ordering form by filling out all the required fields on the form, selecting the pizza size and then finally adding one or more toppings.

I’ve been through the source code and added a few data-cy attributes to the elements referenced inside the e2e test.

Step 9: Run the single E2E test by issuing the following command:

yarn run e2e

Great! Our E2E test successfully passes, verifying that the app functions well and can fill the pizza order form, select the pizza size and add one or more toppings.

E2E tests are sufficient in testing the overall Web app. However, wouldn’t it be beneficial and more effective to complement the E2E tests with visual ones? This adds the value of detecting any visual change or difference upon running E2E regression testing.

Steps 10-11: Add Applitools SDK

Since Applitools integrates seamlessly with Cypress.io, let’s add the Applitools Cypress NPM package to the project.

Step 10: Add the Applitools Eyes Cypress SDK package to the project.

yarn add @applitools/eyes-cypress -D

Step 11: Configure the Applitools Eyes Cypress SDK by issuing the following command:

npx eyes-setup

Now that Applitools Eyes Cypress SDK is configured, we can start using it in our tests.

Let the visual testing begin!

Steps 12-13: Run Test With Applitools To Capture Visual Data

Step 12: Revisit the cypress\integration\pizza.spec.js file and replace its content with:

View the code on Gist.

The code is self-documented.

To integrate Applitools Eyes Cypress SDK into a spec file, you’ll follow the Workflow below:

Start a new test

cy.eyesOpen({

appName: ‘…’,

testName: ‘…’,

browser: { … },

});

Take a snapshot (You may repeat this step wherever you want to take a snapshot)

cy.eyesCheckWindow(‘…’);

End the test

cy.eyesClose();

Step 13: Check the test run in Applitools Test Manager:

Clicking on the test name (in this case, Fill form to order pizza) yields a single snapshot that the Eyes SDK has taken during the running of the test.

The snapshot is labeled full page. The image shows the pizza order form is populated, a pizza size is selected and toppings are selected and showing on top of the pizza.

Since this is the first run of the test, Applitools will save these as the baseline.

Steps 14-16: Demo Applitools With A CSS Change

We will simulate a visual difference by introducing a change on the font-weight for the selected toppings and let Applitools detect this. Now run the test again and see the results.

Step 14: Open the src/app/pizza-app/components/pizza-toppings/pizza-toppings.components.scss file and change the following CSS class to:

input {

border: 0;

clip: rect(0 0 0 0);

height: 1px;

margin: -1px;

overflow: hidden;

padding: 0;

position: absolute;

width: 1px;

}

&–focused {

background: rgba(0, 0, 0, 0.15);

font-weight: 500;

}

Run the test.

Step 15: Issue the command to run the E2E tests again. The test case fails and Applitools detects a visual change for the selected toppings font weight.

Notice how the Applitools Test Manager recorded the second run of the test case and highlighted the visual difference between the two snapshots?

Step 16: Click on the snapshot and compare it to the baseline.

In case you can’t see both snapshots side by side, locate the View menu and select Show both.

To access the source code of this article, feel free to grab a copy of it here: pizza-creator.

Conclusion

You now have some insight on how Applitools speeds up your visual testing experience when you’re developing apps with Cypress and Angular.

My upcoming articles will explore Applitools further, unveil more features and allow you to carry on with E2E and Visual UI Testing easily.

Happy Testing!

For more information on Applitools

The post Cypress and Angular Testing with Applitools appeared first on Automated Visual Testing | Applitools.

Visual testing for React and Storybook

Bilal Haidar — Wed, 17 Jul 2019 01:00:43 +0000

ReactJS provides a library of reusable components for building JavaScript applications. When combined with Storybook and Applitools Eyes, you can validate the visual appearance of your React components in a controlled testbed. With Applitools Ultrafast Grid, you can run your visual validations across a range of browsers and viewport sizes across multiple screen combinations simultaneously. Together, React, Storybook, and Applitools make you much more productive by developing once in React, testing once in Storybook, and rapidly validating everywhere in Applitools.

React Development and Test Best Practices

When developing a React UI component library, you are likely aware that a well-known best practice splits the components into self-contained js/jsx files. Splitting benefits you, whether you are the developer developing the components or another developer consuming those components in your apps.

Once you develop using standalone components, you need to consider how to test your components inside a browser during the development phase. You have a few options here. One of them is to develop a demo or showcase application. You can use the showcase app to render components and make sure they run correctly inside the browser. However, you end up coding something you likely won’t use again – and that seems like a waste of resources.

The best option is to make use of a third-party playground framework such as Storybook that can kick off a new web server for you and run all the components under development. Also, Storybook gives you the option to generate a static documentation playground website that you can host online to showcase your components live.

As you develop components, you revise components, change their code, and test them to verify nothing broke with the new changes. Some of your changes will be functional, and you can create new functional tests for those components. However, the visual changes are problematic. Manual validation of all of them is cumbersome, time-consuming, and inaccurate. That’s where visual UI testing frameworks come in handy.

So, let’s talk Applitools. Not only does Applitools provide you with the means to do Visual UI testing, but you can also use for Visual UI Regression testing. That is heavenly in terms of Visual UI testing.

In this article, I’ll walk you through a complete, step by step guide to visually test a React app with Storybook and Applitools.

What is Applitools?

If you’re not already familiar, Applitools is an automated visual regression testing framework. It focuses on the visual aspects of your app and plays a significant role in exposing the visual differences between baseline snapshots and both current and future snapshots.

Applitools integrates with dozens of development and testing frameworks, such as Cypress.io, Storybook, and Selenium. It provides SDKs for use in your projects to seamlessly communicate and interact with Applitools, to capture both baseline and test screenshots.

So, let’s get started with some visual UI testing. Your visual testing project will depend on React, Storybook, and Applitools. You will also need node.js tools. So, firstly, make sure you’re familiar with the following and have them installed on your machine:

Before we immerse ourselves in writing code, let’s look at how Applitools and Storybook work together.

How Applitools Works With Storybook

No matter whether you use Vue.js, React, or Angular, Storybook doesn’t offer any extension points for third-party frameworks, such as Applitools, to integrate with. The Applitools team worked around this limitation by providing the Applitools Eyes Storybook SDK.

The behavior of this SDK is that once you run it locally on your machine, it will start by collecting all stories in the application, run them all via the Storybook engine, generate DOM snapshots for each and every story and upload these snapshots to the Applitools Eyes server. Here’s a visual of this flow:

Tests run on the server to generate actual image snapshots. Each snapshot is compared to a baseline snapshot (if one exists), and the results are displayed on the Test Manager Dashboard.

This kind of integration provides a smooth experience in terms of writing Storybook stories. Nothing changes in the way you write your React apps.

Now, let’s get on with it and start coding …

Step By Step Guide

For this demonstration of visual testing a React application with Storybook and Applitools, I’ve chosen the React95 open-source library hosted on GitHub under this repo. This library is both unique and fun at the same time. It creates a rich set of components having the theme of Windows 95. This library already uses Storybook and by adding Applitools to the mix, should be much easier and less time-consuming.

Let’s follow the steps to run the React95 Storybook stories and use Applitools to generate a few visual UI test runs.

Steps 1-3 – Start with a React Library and Install Storybook

Step 1: Clone the React95 GitHub repo locally on your machine by running this command:

git clone git@github.com:React95/React95.git

Step 2: Install all NPM package dependencies by running this command:

npm install

Step 3: As you now know, this library already uses Storybook. Hence, the scripts section of the package.json file already contains the script to run Storybook. Let’s give it a shot and run Storybook stories by issuing this command:

npm run storybook

As a side note, in case you are installing Storybook from scratch, here’s a mini-guide to help you get started adding Storybook to your application.

Run the following command to install Storybook into your application:

npx -p @storybook/cli sb init

The Storybook NPM package {{ link to the NPM package here }} has a smart engine behind it that automatically detects the current development framework and installs the correct version of Storybook.

Then to test the Storybook installation is successful, you may run this command:

npm run storybook

For more information on getting started with Storybook and React, visit this online document at https://www.learnstorybook.com/react/en/get-started.

Step 4 – Install Applitools

Step 4: Now that Storybook is running let’s install Applitools. Applitools offers the Applitools Eyes SDK for Storybook NPM package. Let’s add this package to our application by running this command:

npm i @applitools/eyes-storybook

In order to authenticate via the Applitools server, you need to supply the Eyes-Storybook SDK with the API key you got from Applitools. Read more about how to obtain the API key here.

To do this, set the environment variable APPLITOOLS_API_KEY to the API key before running your tests. For example, on Linux/Mac:

export APPLITOOLS_API_KEY=

And on Windows:

set APPLITOOLS_API_KEY=

Step 5 – Validate using Applitools

Step 5: Let’s run the first cycle of the visual UI testing by issuing this command:

npx eyes-storybook

Once the tests are done running, let’s switch back to the Applitools Test Manager Dashboard to check and verify the results.

The left side of the Dashboard lists all batch test runs that you have previously performed. In the above screen capture, the single test run shows the status Passed. In handling this first run, the Applitools engine automatically passes this test and all snapshots thereafter as baseline snapshots.

Click on the single batch test to view the snapshots available. For each and every Storybook story, a snapshot is generated by Applitools.

Let’s take, for example, the Modal story test run. Locate the Modal story test run, click on the snapshot and expand the Window to view the snapper on a bigger scale. The expand button is available at the top-right corner of the Window as shown:

As you can tell, the test run snapshot captures an image snapshot for the story that runs inside the Storybook playground application.

Let’s switch gears and explore a unique gem of Applitools that is the Regression UI testing. Traditional regression testing validates that recent source code changes, deletions, or additions result in behavior that continues to align with expectations.

Applitools allows you to perform visual UI Regression testing. By using it, you can run multiple cycles of test runs and all visual changes collected from one to another, are detected by Applitools servers and logged into the Dashboard.

Steps 6-7 – Create And Validate A Change

Step 6: Let’s introduce a visual change in the Modal component and run our tests again.

Navigate to /components/Modal/Modal.jsx file and change the background-color of the Title Bar to yellow:

const TitleBar = styled.div`

  height: 18px;

  margin-bottom: 2px;

  background-color: yellow;

  color: white;

  padding: 2px 2px 0;

  display: flex;

`;

The change causes the title bar of the Modal window to change color to yellow.

Step 7: Let’s run the second cycle of the visual UI testing by issuing this command:

npx eyes-storybook

Notice the logs on the terminal windows after running the command?

Modal: default [1024x768] - Failed 1 of 1

A total of 1 difference was found.

One of the tests failed and that’s exactly what we expect. Applitools detected the change we introduced to the background color of the Title bar and reported that accordingly.

Let’s switch back to the Applitools Test Manager Dashboard to check the results.

Applitools marks this new test run as Unresolved. Expand on the Modal test run to view the visual differences highlighted to you by Applitools:

Make sure to select the option to view both snapshots so that you can easily view the differences. To do so, locate the following menu and choose the option Both as shown in the figure:

Any test run that results in status Unresolved requires someone’s manual intervention to validate or reject the change. In this case, you may accept the new change(s) and thus update the baseline snapshots.

With an undesirable change, you infer a bug in the source code. You can reject this new test run and keep the original baseline snapshot without any changes.

See The Video Demo

Check out all the steps above in this step-by-step video of how to test React and Storybook:

Want To Know More?

Read about Angular and Storybook with Applitools.

How to troubleshoot and fix React bugs fast [step-by-step tutorial]

Find out more about Applitools. Request a demo, sign up for a free account, or check out our tutorials.

The post Visual testing for React and Storybook appeared first on Automated Visual Testing | Applitools.

React, Cypress.io, and Applitools for Functional and Visual Testing

Bilal Haidar — Wed, 05 Jun 2019 18:52:46 +0000

ReactJS provides a great library for app JavaScript development. Cypress is a great app functional UI test framework. Add Applitools, and you gain productivity in visual testing that allows you to deliver visual testing your app across a range of browsers and operating systems. Put them together, and React plus Cypress plus Applitools is a great combination to develop, functionally test, and visually test your application.

You are reading this article because you either use ReactJS today or have considered using the React library for building front-end components using JavaScript. And, while I assume you are using Cypress.io for functional testing, you could just as easily be using a different toolset and are looking at Cypress asking, “What’s different?”

And, of course, the big reason you are reading this article is to answer the question, “What tools out there give me the productivity gains for visual testing cross-platform that I get when I develop across multiple browser-OS combinations with React and Cypress?” You want to know about Applitools.

Applitools for Visual Test Productivity

Applitools offers a test framework to automate visual testing of your application. It focuses on the visual aspects of your app and plays a major role in exposing the visual differences between baseline snapshots and both current and future snapshots.

Applitools integrates with dozens of testing frameworks, such as Cypress.io, Storybook, and Selenium. It provides SDKs for use in your projects to seamlessly communicate and interact with Applitools.

In my previous post, I showcased the beauty of the Applitools Ultrafast Grid system and demonstrated how Applitool integrates fluidly with Storybook inside an Angular app. With all the attention on React visual UI testing, I thought it is appropriate to share step by step tutorial on this topic. Specifically, React UI testing using Applitools and Cypress.io.

First, ensure that these are properly installed on your machine:

Before we delve into writing code, let’s take a behind-the-scenes look at how Applitools and Cypress.io integrate together.

Applitools / Cypress.io Integration

Applitools integrates with Cypress.io and offers a general-purpose library that can be embedded within your project, whether it be the Angular, Vue, or React app.

The Applitools Eyes Cypress SDK is a simple plugin to Cypress. Once installed, it adds a few commands to the main cy object. Specifically, it adds three main methods: cy.eyesOpen to start the test, cy.eyesCheckWindow to take screenshots (for each test step), and cy.eyesClose to close the test. Here’s a diagram:

Now that you understand how the Applitools/Cypress.io integration works, let’s switch gears and start coding a few visual tests for a React app using Applitools and Cypress.io.

Source code

For this article, I’ve chosen to write a few E2E Cypress.io tests against an open source project written in React JS. The Product Compare app allows the user to select a couple of products and compare them side by side.

Demo Steps 1-5: Get Started

I will take you on a series of steps to install Applitools and Cypress.io on an existing React JS app, add a Cypress.io test, and finally, verify the results on the Applitools Test Manager.

Step 1: Clone the repository locally by issuing the following git command:

git clone git@github.com:Rhymond/product-compare-react.git

Step 2: Install all the npm dependencies by issuing the following command:

npm install

Step 3: Run the app:

npm run start

And you should see something like this:

Voila. It works!

Step 4: Add Cypress package to the project:

npm install --save-dev cypress

The Cypress npm package adds a set of test files to help you write your own automated tests.

Step 5: Open the Cypress Dashboard by issuing the following command:

node_modules/.bin/cypress open

This command opens the Cypress Dashboard and also creates a cypress.json file and a cypress folder in your app’s root directory. The cypress folder is where we will be writing our tests.

Demo Steps 6-8: Add Cypress Tests

Step 6: Add a new, shorter script inside the package.json file to open Cypress Dashboard without having to type that long command you used in the previous step:

"cypress": "cypress open"

In addition, inside the cypress.json file in the root of the application folder, add the following:

{

     "baseUrl": "http://localhost:3000"

}

Now that Cypress is running properly, let’s add our first E2E test.

Step 7: Inside the cypress\integration folder, create the compare.spec.js file and paste the following:

View the code on Gist.

The single E2E test focuses on testing the functionality of adding two products to compare them side by side. There are many tests that can be written, but for our sake, we will play around with a single E2E test to demonstrate the integration of Applitools with Cypress.io.

Now, switch back to the Compare\index.js and Product\index.js files and go through all the data-* attributes I’ve added to facilitate writing this E2E test.

Step 8: Run the single E2E test by issuing the following command:

npm run cypress

Great! Our E2E test successfully passes, verifying that the app functions well and can select multiple products to compare them without any issues.

As you already know, Applitools integrates seamlessly with Cypress.io, so let’s add the Applitools Cypress NPM package to the project.

Demo Steps 9-12: Install and integrate Applitools Eyes

Step 9: Add the Applitools Eyes Cypress SDK package to the project.

npm install @applitools/eyes.cypress --save-dev

Step 10: Configure the Applitools Eyes Cypress SDK by issuing the following command:

npx eyes-setup

Now that Applitools Eyes Cypress SDK is configured, we can start using it in our tests.

Let the visual testing begin!

Step 11: Revisit the cypress\integration\compare.spec.js file and replace its content with the following:

View the code on Gist.

The code is self-documented.

To integrate Applitools Eyes Cypress SDK into a spec file, you’ll follow the Workflow below:

Start a new test

cy.eyesOpen({

   appName: '...',

   testName: '...',

   browser: { ... },

});

Take a snapshot (You may repeat this step wherever you want to take a snapshot)

cy.eyesCheckWindow('...');

End the test

cy.eyesClose();

Notice the Applitools assertions at lines 1, 7, 12, and 17? This type of logging informs you about the steps Applitools is taking while Cypress runs your tests.

Step 12: Check the test run in Applitools Test Manager:

Clicking on the test name (in this case, Select 2 products to compare) yields the two snapshots that the Eyes SDK has taken during the running of the test.

The first snapshot is labeled first product selected. The image shows the Cherry product selected.

The second snapshot is labeled second product selected and shows the Orange product selected.

Since this is the first run of the test, Applitools will save these as the baseline.

We will simulate a visual difference by introducing an underline to all prices displayed on the page and let Applitools detect this. Now run the test again and see the results.

Demo Steps 13-15: Introduce Changes and Validate

Step 13: Open the src/components/Product/styles.css file and change the following CSS class as follows:

.product .stats-container .product_price {

   float: right;

   color: #48cfad;

   font-size: 22px;

   font-weight: 800;

   text-decoration: underline;

}

Let’s run the test.

Step 14: Issue the command to run the E2E tests again. The test case fails, and Applitools detects a visual change for the product image overlays.

Notice how the Applitools Test Manager recorded the second run of the test case and highlighted the visual difference between the two snapshots?

Step 15: Click on the first snapshot and compare to the baseline.

In case you can’t see both snapshots side by side, locate the View menu and select Show both.

In addition to the capacity to visually test our React JS apps, Applitools offers more functions and features to give you the upper hand in testing your web apps quickly and locate the visual differences easily.

To access the source code of this article, feel free to grab a copy of it here: product-compare-react.

Conclusion

You now have some insight on how Applitools speeds up your visual testing experience when you’re developing apps with React JS and Cypress.

In upcoming articles, I will explore Applitools further, unveil more features, and give you the power to carry on with E2E and Visual UI Testing easily.

Happy Testing!

Some Of My Recent Posts

Here are some of the previous Applitools topics I’ve discussed:

For More Information From Applitools

To read more about Applitools’ visual UI testing and Application Visual Management (AVM) solutions, check out the resources section on the Applitools website. To get started with Applitools, request a demo or sign up for a free Applitools account.

The post React, Cypress.io, and Applitools for Functional and Visual Testing appeared first on Automated Visual Testing | Applitools.

Visually Test Vue.js application using Applitools and Storybook [step-by-step tutorial]

Bilal Haidar — Sat, 27 Apr 2019 00:01:05 +0000

So, you want to know about visual testing for your Vue.js app in Storybook using Applitools? So did I!

In this article, I’ll walk you through a complete, step-by-step guide of how to visually test a Vue.js app with Storybook and Applitools.

So, let’s get started with some visual UI testing. Your visual testing project will depend on Vue.js, Storybook, and Applitools. You will also need the node.js tools. So, firstly, make sure you’re familiar with the following and have them installed on your machine:

Before we immerse ourselves in writing code, let’s look at how Applitools and Storybook work together.

How Applitools works with Storybook

The behavior of this SDK is that once you run it locally on your machine, it will first start by collecting all stories in the application, run them all via the Storybook engine, generate DOM snapshots for each and every story, and finally upload these snapshots to the Applitools Eyes server. Here’s a visual of this flow:

On the server, the tests are run and an actual image snapshot is generated. Each and every snapshot is compared to a baseline snapshot (if one exists) and results are displayed on the Test Manager Dashboard.

This kind of integration provides a smooth experience in terms of writing Storybook stories. Nothing changes in the way you write your Vue apps.

Now, let’s get on with it and start coding …

Step by Step Guide

For this demonstration of visual testing a Vue.js application with Storybook and Applitools, I’ve chosen the TodoMVC example, published as part of the examples projects on the Vue JS website.

Let’s follow the steps to run the TodoMVC app and add a few Storybook stories.

Step 1: Install the Vue CLI 3 on your machine by issuing this command:

npm install -g @vue/cli

# OR

yarn global add @vue/cli

Once the Vue CLI is installed, you can start creating the Vue JS app.

Step 2: Clone the Vue.js TodoApp app locally by issuing this command:

git clone git@github.com:bhaidar/todomvc-vue.git

Step 3: Install all the npm dependencies. Issue the following command:

yarn

Step 4: Checkout the Vue.js/Storybook branch. Enter:

git checkout --track origin/vuejs-testing-with-storybook

Step 5: Run the app.

yarn serve

And you should see something like this:

So, it works!

Step 6: To add the Storybook package to a Vue.js app, you issue the following command:

npx -p @storybook/cli sb init --type vue

The command takes care of automatically installing and configuring Storybook folders and files to your project. It even adds an NPM script in the Package.json file to make it easy to run Storybook stories.

There are two methods to set up Storybook in a Vue.js app. In our case, we’ve chosen the automatic setup. You can read more about setup modes on Storybook for Vue.

There is no need to install Storybook on this project. I’ve already taken care of that for you.

Step 7: Run the default stories added by Storybook by issuing this command:

yarn storybook

It opens the Storybook Dashboard to start running your stories.

Step 8: Refactor the code and extract a new Todo component. In general, and especially when writing Storybook stories, it’s recommended that you structure your code into a set of components that could exist on their own. Check the new Todo component on the repo:

View the code on Gist.

Step 9: Add a new Storybook story to test the Todo component. Locate the “./stories/index.stories.js” file and replace the content with the following:

View the code on Gist.

The code adds a default story to test the state of the Todo component. When defining Storybook stories for Vue.js, the callback function of the stories.add() function should return a Vue instance. Think of this Vue instance as a host component with a template to run the Todo component. You can customize this host Vue instance the way you want, using the standard Vue component methods and properties.

One important thing to note in relation to how a Storybook story communicates with a Vue.js component. Notice the code definition of the data () method that is defined on the host Vue instance? Since the Todo component expects two properties, you need to define these two properties inside the data() method and attach those properties on the Todo component inside the HTML template.

Now that the story is ready, let’s run it with Storybook by issuing this command:

yarn storybook

Step 10: Install the Applitools Eyes SDK for Storybook NPM package by issuing the following command:

npm i @applitools/eyes-storybook

Step 11: Run it with the command:

npx eyes-storybook

Make sure you get an Applitool API Key and store it on your machine. For a complete tutorial on how to install and run Applitools Storybook SDK for Angular, check out this article: Storybook Angular Tutorial.

The command simply opens the Storybook stories, runs them one by one, and then sends all the DOM snapshots to the Applitools Ultrafast Grid Server.

If you expand the single test run, you get the following result:

You can see how simple it is to add and configure Applitools Storybook SDK into your existing Vue.js application.

Let’s continue on and introduce a visual change and run our UI tests once again.

Step 12: Introduce a background color change to the header of the main input field on the page.

Open the /src/components/Todo.vue component and apply this change:

View the code on Gist.

Step 13: Run the Eyes Storybook SDK once again with this command::

npx eyes-storybook

Looking at the test results on the Test Manager, you can see how Applitools captured the visual difference and marked the test results as a Mismatch.

So far, we’ve been running Applitools with Storybook using the default test configuration settings. Let’s see how we can customize the Applitools Eyes Storybook test configurations and run our Storybook stories with a variety of custom viewports.

Visual UI tests with custom viewports

Keeping in mind that visual testing Vue.js with Storybook and Applitools doesn’t need any kind of configuration, the important test configurations are auto-inferred by Applitools. However, if you want to play around with the test configuration, you have two options to specify:

Environment variables
The applitools.config.js file

When you’re using Applitools with Storybook, there’s no need to add any specific Applitools commands or set test configuration to enable visual testing.

In my previous article on How I ran 100 UI tests in just 20 seconds you learned that Applitools Eyes SDK runs the Storybook stories, takes a snapshot of each, and sends the DOM snapshots to the Ultrafast Grid backend server. It saves the test run with the default test configuration.

Step 1: Add a new JavaScript file to the root of your application. Give it a name of applitools.config.js.

Step 2: Add the appName and batchName configuration settings as follows:

View the code on Gist.

Step 3: Run the test case on two different browsers with two different viewports by adding the following:

View the code on Gist.

Step 4: Save and run the tests again by issuing the following command:

npx eyes-storybook

Step 5: Check the test run in Applitools Test Manager.

I’ve switched the Test Manager view to the Batch Details View.

Running this test yields an execution time of just two seconds! To be clear, this is what we spent rendering baseline images, once the DOM snapshots were uploaded. Even so, that’s really fast.

It’s the duty of the Ultrafast Grid to analyze the test configuration file attached to the test and spawn, in parallel, a number of browser instances to handle the test run efficiently.

Tweaking the Concurrency

Applitools Eyes SDK gives testers as well as developers the upper hand in configuring many aspects of the test configuration settings including the Concurrency field. By setting the value of this field, you can instruct the Ultrafast Grid component to use a certain number of parallel threads or processes in order to run and your visual UI tests.

Let’s add more custom viewports into test configuration settings and also set the Concurrency field to a value of 100 and see the results.

Step 1: Replace the content of the applitools.config.js file with the following configuration:

View the code on Gist.

In this test configuration, we are running 20 tests with 10 unique view ports on two major browsers: Firefox and Google Chrome.

Step 2: Save and run the tests by issuing the following command:

npx eyes-storybook

The Eyes SDK/Server took only 15 seconds to complete all the above tasks, create baselines, and report the results on Applitools Test Manager Dashboard, for the entire batch of 20 test runs for a single Storybook story.

Step 3: Check the test run in Applitools Test Manager.

I’ve switched the Test Manager view to the Batch Summary View.

Let’s see the power of the Eyes server when running a regression testing cycle.

Step 4: Introduce a background color change to the header of the main input field on the page.

Open the /src/components/Todo.vue component and apply this change:

View the code on Gist.

Step 5: Run the Eyes Storybook SDK once again with this command::

npx eyes-storybook

Once again, the Eyes SDK/Server took only 17 seconds to upload the DOM snapshot, run the entire batch of visual UI tests, and display the results on the Applitools Test Manager Dashboard.

Step 6: Check the test run in Applitools Test Manager.

The Test Manager highlights the differences between the baseline snapshot and the current test run and displays a mismatch icon near the snapshots.

At this moment, you can either accept the new changes and make them the new baseline or reject them and keep the old baseline.

Once again, it takes the power of the Ultrafast Grid to analyze the test configuration file and set up the testing environment as needed.

In conclusion

Whether you are developing in Angular, Vue.js, React, or any other technology of choice, Applitools have you covered, in running your visual UI tests, by providing a rich set of SDKs to cover many test frameworks out there!

Happy Testing!

Find out more about Applitools. Setup a live demo with us, or if you’re the do-it-yourself type, sign up for a free Applitools account and follow one of our tutorials.

Bilal Haidar, Author at Automated Visual Testing | Applitools

How Visual UI Testing can speed up DevOps flow [Revisited]

Source Code Repository Management System

Continuous Integration / Continuous Delivery (CI/CD)

Visual Testing Workflow

Demo

1. Cloning the example code

2. Configuring Applitools API Key

3. Connecting the GitHub repo with CircleCI

3a. Build in GitHub

3b. Integrate with Applitools

3c. Push Changes

4. Configuring the GitHub repo

5. Connecting the GitHub repo with Applitools

6. Running a visual UI test locally

7. Running the Visual UI testing workflow

Conclusion

Other Related Content

How I ran 100 UI tests in just 20 seconds [Revisited]

Life before Ultrafast Grid

Life with Ultrafast Grid

Source code

Applitools Ultrafast Grid Demo

Visual UI tests with custom viewports

Visual UI tests with device emulation

100 Visual UI tests

See for yourself

Conclusion

Visual Testing with Applitools, Appium, and Amazon AWS Device Farm

AWS Device Farm for mobile visual testing

Selenium Webdriver for browser app automation

Appium for mobile app automation

Demo

Prerequisites

Component Installations

Run the Visual UI Test Script locally

Verify Test Results on Applitools Test Manager

Run the Visual UI Test Script on AWS Device Farm

Verify Test Results on Applitools Test Manager

Conclusion

For More Information

How Visual UI Testing can speed up DevOps flow

Source Code Repository Management System

Continuous Integration / Continuous Delivery (CI/CD)

Visual Testing Workflow

Demo

1. Cloning the example code

2. Configuring Applitools API Key

3. Connecting the GitHub repo with CircleCI

3a. Build in GitHub

3b. Integrate with Applitools

3c. Push Changes

4. Configuring the GitHub repo

5. Connecting the GitHub repo with Applitools

6. Running a visual UI test locally

7. Running the Visual UI testing workflow

Conclusion

Other Related Content

Cross-browser Testing With Cypress.io

Welcome to Cypress.io

Show me an E2E test

Oh, Cypress.io only works with Google Chrome!?!

Applitools to complement Cypress.io

Applitools Ultrafast Grid – Setup Code

Applitools Ultrafast Grid – Snapshot Capture

Applitools Ultrafast Grid – Grid Tests

Applitools/Cypress.io Integration

Cross-browser testing step by step guide

Create a new project folder

Install Cypress.io

Write your first E2E test

Run the test

Install Applitools Eyes Cypress SDK

Configure Applitools API Key

Write your first Visual test

Run the test

Verify Test Results

Conclusion

Other Related Content

How Cross-Browser Testing Is Evolving