Test Automation University (also called “TAU”) is one of the best online platforms for learning testing and automation skills. TAU offers dozens of courses from the world’s leading instructors, and everything is available for free. The platform is proudly powered by Applitools. As of November 2022, nearly 140,000 students have signed up! TAU has become an invaluable part of the testing community at large. Personally, I know many software teams who use TAU courses as part of their internal onboarding and mentorship programs.

So, which TAU courses are currently the most popular? In this list, we’ll count down the top 10 most popular courses, ranked by the total number of course completions over the past year. Let’s go!

#10: Selenium WebDriver with Java

Starting off the list at #10 is Selenium WebDriver with Java by none other than Angie Jones. Even with the rise of alternatives like Cypress and Playwright, Selenium WebDriver continues to be one of the most popular tools for browser automation, and Java continues to be one of its most popular programming languages. Selenium WebDriver with Java could almost be considered the “default” choice for Web UI test automation.

In this course, Angie digs deep into the WebDriver API, teaching everything from the basics to advanced techniques. It’s a great course for building a firm foundation in automation with Selenium WebDriver.

#9: Python Programming

#9 on our list is one of our programming courses: Python Programming by Jess Ingrassellino. Python is hot right now. On whatever ranking, index, or article you find these days for the “most popular programming languages,” Python is right at the top of the list – often vying for the top spot with JavaScript. Python is also quite a popular language for test automation, with excellent frameworks like pytest, libraries like requests, and bindings for browser automation tools like Selenium WebDriver and Playwright.

In this course, Dr. Jess teaches programming in Python. This isn’t a test automation course – it’s a coding course that anyone could take. She covers both structured programming and object-oriented principles from the ground up. After two hours, you’ll be ready to start coding your own projects!

#8: API Test Automation with Postman

The #8 spot belongs to API Test Automation with Postman by Beth Marshall. In recent years, Postman has become the go-to tool for building and testing APIs. You could almost think of it as an IDE for APIs. Many test teams use Postman to automate their API test suites.

Beth walks through everything you need to know about automating API tests with Postman in this course. She covers basic features, mocks, monitors, workspaces, and more. Definitely take this course if you want to take your API testing skills to the next level!

#7: Introduction to Cypress

Lucky #7 is Introduction to Cypress by Gil Tayar. Cypress is one of the most popular web testing frameworks these days, even rivaling Selenium WebDriver. With its concise syntax, rich debugging features, and JavaScript-native approach, it’s become the darling end-to-end test framework for frontend developers.

It’s no surprise that Gil’s Cypress course would be in the top ten. In this course, Gil teaches how to set up and run tests in Cypress from scratch. He covers both the Cypress app and the CLI, and he even covers how to do visual testing with Cypress.

#6: Exploring Service APIs through Test Automation

The sixth most popular TAU course is Exploring Service APIs through Test Automation by Amber Race. API testing is just as important as UI testing, and this course is a great way to start learning what it’s all about. In fact, this is a great course to take before API Test Automation with Postman.

This course was actually the second course we launched on TAU. It’s almost as old as TAU itself! In it, Amber shows how to explore APIs first and then test them using the POISED strategy.

#5: IntelliJ for Test Automation Engineers

Coming in at #5 is IntelliJ for Test Automation Engineers by Corina Pip. Java is one of the most popular languages for test automation, and IntelliJ is arguably the best and most popular Java IDE on the market today. Whether you build frontend apps, backend services, or test automation, you need proper development tools to get the job done.

Corina is a Java pro. In this course, she teaches how to maximize the value you get out of IntelliJ – and specifically for test automation. She walks through all those complicated menus and options you may have ignored otherwise to help you become a highly efficient engineer.

#4: Java Programming

Our list is winding down! At #4, we have Java Programming by Angie Jones. For the third time, a Java-based course appears on this list. That’s no surprise, as we’ve said before that Java remains a dominant programming language for test automation.

Like the Python Programming course at spot #9, Angie’s course is a programming course: it teaches the fundamentals of the Java language. Angie covers everything from “Hello World” to exceptions, polymorphism, and the Collections Framework. Clocking in at just under six hours, this is also one of the most comprehensive courses in the TAU catalog. Angie is also an official Java Champion, so you know this course is top-notch.

#3: Introduction to JavaScript

It’s time for the top three! The bronze medal goes to Introduction to JavaScript by Mark Thompson. JavaScript is the language of the Web, so it should be no surprise that it is also a top language for test automation. Popular test frameworks like Cypress, Playwright, and Jest all use JavaScript.

This is the third programming course TAU offers, and also the top one in this ranking! In this course, Mark provides a very accessible onramp to start programming in JavaScript. He covers the rock-solid basics: variables, conditionals, loops, functions, and classes. These concepts apply to all other programming languages, too, so it’s a great course for anyone who is new to coding.

#2: Web Element Locator Strategies

I’m partial to the course in second place – Web Element Locator Strategies by me, Andrew Knight! This was the first course I developed for TAU, long before I ever joined Applitools.

In whatever test framework or language you use for UI-based test automation, you need to use locators to find elements on the page. Locators can use IDs, CSS selectors, or XPaths to uniquely identify elements. This course teaches all the tips and tricks to write locators for any page, including the tricky stuff!

#1: Setting a Foundation for Successful Test Automation

It should come as no surprise that the #1 course on TAU in terms of course completions is Setting a Foundation for Successful Test Automation by Angie Jones. This course was the very first course published to TAU, and it is the first course in almost all the Learning Paths.

Before starting any test automation project, you must set clear goals with a robust strategy that meets your business objectives. Testing strategies must be comprehensive – they include culture, tooling, scaling, and longevity. While test tools and frameworks will come and go, common-sense planning will always be needed. Angie’s course is a timeless classic for teams striving for success with test automation.

What can we learn from these trends?

What other courses are popular?

The post The Top 10 Test Automation University Courses appeared first on Automated Visual Testing | Applitools.

Jenkins is a popular open source automation server. It’s used to implement Continuous Integration (CI) and Continuous Delivery (CD) for any development project.

CI/CD, a key component of a DevOps strategy, allows you to shorten the development lifecycle while maintaining quality by automating tasks like testing. In short, a successful implementation will help you test more often and deploy faster with high quality. 

Jenkins, as a Java-based CI server with strong community support and a huge plugin ecosystem, is a powerful tool for anyone looking to add CI/CD to their project.

History of Jenkins

Jenkins has a long history, stretching back to 2011, and its roots extend back even further to 2004. That was the year that Kohsuke Kawaguchi, a Java developer at Sun Microsystems, built an automation tool to help him answer the question of whether his code would break the build before he committed it. The open source tool he built, called Hudson, automated tests and builds and helped him understand whether his code would work much sooner. 

In 2011, there was an issue between Oracle (which had acquired Sun Microsystems) and the open source project’s community, and the community created a fork named Jenkins. For a while both projects were updated independently, but Jenkins proved to be more popular and Hudson is no longer developed. 

What Does Jenkins Do?

Jenkins helps you automate tasks for CI/CD. Once the Jenkins server is configured, you will be able to execute a series of automated tests and builds so that your code is always up to date and valid. Implementing CI/CD through a tool like Jenkins can greatly simplify the process of ensuring a high level of code quality and successful builds. It’s particularly powerful when large development teams are working on a single project, as traditional methods can result in a lot of conflicting code commits that require complex troubleshooting.

Before we dive further into the functionality of Jenkins, let’s take a moment to define continuous integration/delivery. Then we’ll discuss how Jenkins helps you achieve it.

What is Continuous Integration and Continuous Delivery (CI/CD)?

CI is a process that enables you to integrate code changes from multiple developers working on a single project quickly and repeatedly. When a developer commits code, it can immediately be tested. If the tests pass, the code can be integrated into the build, which can similarly be tested and immediately verified. 

Automated tests are typically used alongside automated builds so that the process can be fast, effective and easily repeatable.

How Does Jenkins Help with Automated Testing?

Using Jenkins to implement a CI/CD process can dramatically improve your ability to test and ship a high quality product. 

One of the chief benefits is the ability to rapidly discover and fix bugs. If newly committed code introduces a defect into the build, not only is it caught immediately, but it is easy to know whose code caused the error. The problematic code can then be isolated, updated and recommitted quickly. 

Getting Started with Jenkins and Jenkins Pipeline

The Jenkins CI server can be run as a standalone application with a built-in Java servlet container, or as a servlet in other Java servlet containers such as Apache Tomcat.

The most popular way to manage CI/CD with Jenkins is through Jenkins Pipeline, which is a suite of plugins that help you integrate continuous delivery pipelines in Jenkins. Essentially, a Pipeline is a linked series of automated steps to be executed by Jenkins. Jenkins Pipelines are implemented in code which can be committed, so that it can be versioned and reviewed along with the rest of your code.

Here’s an example (from the Jenkins docs) of a CD scenario modeled in Jenkins Pipeline:

Pipelines can be written manually or through the newer Blue Ocean GUI. There is also an older “classic UI”, but if you want to use a UI Blue Ocean is recommended.

Extending Jenkins with Plugins

Jenkins is designed to be easily extended via plugins – and over the years a thriving community has created a huge plugin ecosystem. The strength of this community, and subsequently the size of the plugin library, is one of the best things about Jenkins.

You can find a plugin for just about anything you need, as there are already 1500+ plugins contributed by the community (and counting). This is a good place to note that Jenkins is very actively updated and open to community participation.

Many testing tools like Applitools have a plugin that can easily help get your testing and CI/CD working together. Check out the Applitools plugin for Jenkins to see how this works.

Should You Use Jenkins?

Jenkins is a popular open source tool for CI/CD that is free to use. While you may need some server administration skills to configure and monitor Jenkins, there are many advantages to consider. The Jenkins project includes a large plugin ecosystem, the community around it is thriving and it is actively developed. If that sounds appealing to you, then give Jenkins a look for your CI/CD needs.

Learn More

• Continuous Integration with Jenkins [Free Course]
• How to update Jenkins build status in GitHub pull requests [Step-by-Step Tutorial]
• Jenkins Documentation [official]

The post What is Jenkins? How to Use Jenkins for CI/CD and Testing appeared first on Automated Visual Testing | Applitools.

I took Nyran Moodie’s course on Test Automation University: Automated Visual Testing with WebdriverIO. If you want the explicit link to the course, here it is: https://testautomationu.applitools.com/automated-visual-testing-javascript-webdriverio/index.html

Course Introduction

If you use WebdriverIO regularly, and you are unfamiliar with the basics of using Applitools for automated visual testing, you will appreciate this course. Nyran focuses you on how to add visual test automation with Applitools to your WebdriverIO tests.

Nyran expects you to know WebdriverIO. If you want to learn WebdriverIO, check out Julia Pottinger’s excellent course – UI Automation with WebdriverIO on Test Automation University.

Nyran uses JavaScript for his examples and Visual Studio Code as his IDE. In case you didn’t know, you can use Applitools with a range of test languages. No matter what your test language of choice or IDE, I think you will find Nyran’s code and coding approach fairly intuitive.

Course Structure

Nyran breaks the course into eight main chapters. He gives this description in the course overview:

• 1 – I am going to start by giving you an introduction to visual testing.
• 2 – We are going to look at how we can get our environment set up to start using Applitools eyes.
• 3 – We are going to create and run our first visual test using Applitools
• 4 – I want to introduce you to the different match levels that Applitools has and the concept of viewport sizes
• 5 – I will be talking about checkpoints.
• 6 – I will be looking at how we can organize our visual tests using batches
• 7 – We will look at how we can analyze our test results using the Test Manager.
• 8 – We will be looking at the integration and collaborations that Applitools provides.

Each of these chapters provides a methodical approach to getting going with Applitools. I’ll give a quick overview of each.

Chapter 1 – Why Visual Testing

If you read this review or take the course, you know why visual testing. Your UI and end-to-end tests result in rendered output. You can write all the functional tests that grab locators, enter data, effect action, and cause the appropriate output locators to have the appropriate values. But, until you actually look at the result, you cannot tell if the input and output conform to design and usability expectations.

Nyran did not explain the most frequently-experienced reason for visual testing – unintended consequences of code changes over time. Our experience shows us that most expected application changes get tested, but unintended changes cause problems.

Chapter 2 – Getting Started With Applitools

Nyran does a nice job explaining how to get started. You need an Applitools API key, which you can get from the Applitools console. Nyran explains why you set up a local environment variable for your API key (so you do not need to include your API key in your test code directly). He also points to the github repo he uses for all the examples in the course.

Chapter 3 – Create And Run A Visual Test with Applitools

Chapter 3 involves the first coding examples for setting up Applitools. With a simple:

npm install  @applitools/eyes.webdriverio

You get the the node instructions for installing the Applitools Eyes to your WebdriverIO setup. After this you can install the Applitools Eyes service to your tests. He shows code examples of what test code looks like when calling Applitools:

Once he walks you through a test page example and fills in the tests, he gets code that looks like a full test.

Finally, he shows you the Applitools UI and how it highlights differences found during a test. To do this, he shows a test site with the ability to show different content, and he shows how Applitools highlights the differences.

Nyran makes it clear that Applitools can find and highlight all the visual changes on a comparison page (the checkpoint) versus the original capture (the baseline). And, he explains that Applitools lets you accept the checkpoint as the new baseline, or reject it.

Chapter 4 – Viewports and Match Levels

Nyran breaks chapter 4 into two sections. In Section 4.1, Nyran goes through the idea of viewport sizes. If you build a responsive app, you want to run your app for 4K down to mobile device sizes. How do you validate the different device size views? Applitools makes it easy to add a JavaScript file that specifies all the viewport sizes you want to validate. Applitools runs all the captures for you.

Next, Nyran writes about match levels. Applitools default comparison level, called “strict”, compares visually noticeable changes between a checkpoint and baseline. Strict uses Applitools Visual AI to break items on a page into elements that it then compares.

However, sometimes strict is too strict. In “content” match level, Applitools checks text and structures but ignores color variations. This match level helps when you apply a global change to color and want to ensure that – color outstanding – no other changes have taken place. And, if they have, you want those changes highlighted quickly.

In “layout” match level, Nyran shows, Applitools lets you validate pages that have dynamic content sharing a common structure. For example, you might have a retail shoppping page that shows user-specific and hot items updating from test run to test run. Or, you have a news site that updates top stories regularly. Layout match level pays attention to the layout structure (relationship of sections and text sizes) without comparing the specific contents of any set of elements within that structure.

Chapter 5 – Checkpoints

Nyran spends this chapter reviewing the ways Applitools lets  you make visual captures. First, ou can capture the visible page – the current viisble screen. Alternatively, you can capture the full page. Applitools runs through the app horizontally and vertically and stitches the images into a single screen baseline and checkpoint.

Next, you can capture individual web elements. Finally, you can capture frames on a page. Your flexibility in setting checkpoints gives you plenty of power to control the details of your inspection at different times in your testing life cycle.

Chapter 6 – Batches

Batches provide a way of organizing common tests inside the Applitools Test Manager. Nyran explains how to code tests into batches. He also shows how batched tests get grouped inside the Test Manager. When you use batches, your test results become easier to interpret.

Nyran implies two things about batches. First, grouping tests into batches make your testing much easier to understand. Second, If you want the benefit of batches, you need to code those batch definitions yourself.

Chapter 7 – Using the Test Manager

The Test Manager is the Applitools service user interface. Your user login provides you an API key, and tests run under your API key collect in your view in Test Manager.

Nyran shows you the basics, as well as some cool tricks in the Test Manager. He shows you how to compare differences between the checkpoint and the baseline. You can approve expected changes and update the baseline to the checkpoint. Or, you can reject changes and have them routed back to development as bugs.

Next, Nyran shows you how tests get grouped and how to use statistics. He also shows you how to override the existing match level on all or part of a test. Finally, he shows you Automated Test Maintenance.

Automated Test Maintenance gives you huge powers of scale for validating changes to your application. When you find a change in one checkpoint and approve it as a valid change, Applitools finds all other similar changes and gives you the power to approve those identical changes at the same time. For example – you change your menu bar at the top of your app, and the change affects 145 pages. Following your validation of a change on one page, Applitools asks you if you want to approve the other 144 pages with the identical change. That’s powerful.

Chapter 8 – Integrations

In the last chapter, Nyran shows how Applitools integrates with other tools in development – especially your CICD workflows.

Nyran shows the range of Applitools test framework and language SDKs that work on web and mobile test frameworks. Applitools lets you capture screenshots to use Visual AI if you use an unsupported framework. Applitools even lets you compare PDFs to ensure that your document generators behave correctly even as you update your application.

Next, you see how to link Applitools with Jira. You can link issues found in Applitools with incidents in Jira. Also, you can link GitHub and Applitools to tie your image comparisons to the source code GitHub pull requests. Finally, you see the standard approach to having Applitools link with your favorite CICD workflow manager.

Conclusion

Nyran wrote a nice course on how to use Applitools for automated visual testing. He makes clear that he used WebdriverIO and JavaScript because he knew these well. However, he knows the range of choices available to you.

Nyran really doesn’t cover:

• Setting up tests in WebdriverIO,
• Approaches for running and managing tests and test results
• Managing WebdriverIO in your CICD workflow

You can find other courses to address these issues.I enjoyed taking Nyran’s course. Having taken other JavaScript testing courses, I think Nyran provides good examples of Applitools. He wants you to know how to get the most out of Applitools when you use WebdriveriO.

As always, I include my certificate of completion from Test Automation University:

For More Information

• Visual Testing – FAQ
• Visual Testing: The Missing Piece of the Puzzle – with Gil Tayar
• 20 Highly Qualified Test Automation Superstars
• How Do You Simplify End-To-End Test Maintenance?
• Five Data-Driven Reasons To Add Visual AI To Your End-To-End Tests
• Sign up for a free Applitools account
• Request an Applitools demo

The post Review – Automated Visual Testing With WebdriverIO appeared first on Automated Visual Testing | Applitools.

What does it take to add UI tests in your CICD pipelines?

On March 12, Angie Jones, Senior Developer Advocate at Applitools, sat down with Jessica Deen, Senior Cloud Advocate for Microsoft, held a webinar to discuss their approaches to automated testing and CI. 

Angie loves to share her experiences with test automation. She shares her wealth of knowledge by speaking and teaching at software conferences all over the world, as well as writing tutorials and blog posts on angiejones.tech.

As a Master Inventor, Angie is known for her innovative and out-of-the-box thinking style which has resulted in more than 25 patented inventions in the US and China. In her spare time, Angie volunteers with Black Girls Code to teach coding workshops to young girls in an effort to attract more women and minorities to tech.

Jessica’s work at Microsoft focuses on Azure, Containers, OSS, and, of course, DevOps. Prior to joining Microsoft, she spent over a decade as an IT Consultant / Systems Administrator for various corporate and enterprise environments, catering to end users and IT professionals in the San Francisco Bay Area.

Jessica holds two Microsoft Certifications (MCP, MSTS), 3 CompTIA certifications (A+, Network+, and Security+), 4 Apple Certifications, and is a former 4-year Microsoft Most Valuable Professional for Windows and Devices for IT.

The Talk

Angie and Jessica broke the talk into three parts. First, Angie would discuss factors anyone should consider in creating automated tests. Second, Angie and Jessica would demonstrate writing UI tests for a test application.  Finally, they would work on adding UI tests to a CI/CD pipeline.

Let’s get into the meat of it.

Four Factors to Consider in Automated Tests

Angie first introduced the four factors you need to consider when creating test automation:

• Speed
• Reliability
• Quantity
• Maintenance

She went through each in turn.

Speed

Angie started off by making this point:

“When your team checks in code, they want to know if the check-in is good as quickly as possible. Meaning, not overnight, not hours from now.”

Angie points out that the talk covers UI tests primarily because lots of engineers struggle with  UI testing. However, most of your check-in tests should not be UI tests because they run relatively slowly.  From this she referred to the testing pyramid idea

• Most of your tests are unit tests – they run the fastest and should pass (especially if written by the same team that wrote the code)
• The next largest group is either system-level or business-layer tests. These tests don’t require a user interface and show the functionality of units working together
• UI tests have the smallest number of total tests and should provide sufficient coverage to give you confidence in the user-level behavior.

While UI tests take time, Angie points out that they are the only tests showing user experience of your application. So, don’t skimp on UI tests.

Having said that, when UI tests become part of your build, you need to make sure that your build time doesn’t become bogged down with your UI tests. If all your conditions run over 15 minutes, that’s way too long. 

To keep your testing to a minimum, Angie suggests running UI tests in parallel. To determine whether or not  you need to split up one test into several parallel tests, give yourself a time limit. Let’s say your build needs to complete in five minutes. Once you have a time limit, you can figure out how many parallel tests to set up. Like – with the 15 minute example, you might need to divide into three or more parallel tests.

Reliability

Next, you need reliable tests. Dependable. Consistent. 

Unreliable tests interfere with CI processes. False negatives, said Angie, plague your team by making them waste time tracking down errors that don’t exist. False positives, she continues, corrupt your product by permitting the check-in of defective code. And, false positives corrupt your team because bugs found later in the process interfere with team cohesion and team trust. 

For every successful CICD team, check-in success serves as the standard for writing quality code. You need reliable tests.

How do you make your tests reliable?

Angie has a suggestion that you make sure your app includes testability – which involves you leaning on your team. If you develop code, grab one of your test counterparts. If you test, sit down with your development team. Take the opportunity to discuss app testability.

What makes an app testable? Identifiers. Any test runner uses identifiers to control the application. And, you can also use identifiers to validate outputs. So, a consistent regime to create identifiers helps you deliver consistency. 

If you lack identifiers, you get stuck with CSS Selectors or Xpath selectors. Those can get messy – especially over time.

Another way to make your app testable, Angie says, requires code that lets your test set initial conditions. If your UI tests depend on certain data values, then you need code to set those values prior to running those tests. Your developers need to create that code – via API or stored procedure – to ensure that the tests always begin with the proper conditions. This setup code can help you create the parallel tests that help your tests run more quickly.

You can also use code to restore conditions after your tests run – leaving the app in the proper state for another test.

Quantity

Next, Angie said, you need to consider the number of tests you run.

There is a common misconception that you need to automate every possible test condition you can think about, she said. People get into trouble trying to do this in practice.

First, lots of tests increase your test time. And, as Angie said already, you don’t want longer test times.

Second, you end up with low value as well as high-value UI tests.  Angie asks a question to help triage her tests:

“Which test would I want to stop an integration or deployment? If I don’t want this test to stop a deployment, it doesn’t get automated. Or maybe it’s automated, but it’s run like once a day on some other cycle, not on my CICD.”

Angie also asks about the value of the functionality:

“Which test exercises critical, core functionality? Those are the ones you want in there. Which tests cover areas of my application that have a history of failing? You’re nervous anytime you have to touch that code. You want some tests around that area, too.”

Lastly, Angie asks, which tests provide information already covered by other tests in the pipeline? So many people forget to think about total coverage. They create repetitive tests and leave them in the pipeline. And, as many developers know, a single check-in that triggers multiple failures can do so because it was a single code error that had been tested, and failed, multiple times. 

“Don’t be afraid to delete tests,” Angie said. If it’s redundant, get rid of it, and reduce your overall test code maintenance. She talked about how long it took her to become comfortable with deleting tests, but she appreciates the exercise now. 

Maintenance

“Test code is code,” Angie said. “You need to write it with the same rules, the same guidelines, the same care that you would any production code.”

Angie continued, saying that people ask, “‘Well, Angie, why do I need to be so rigorous with my test code?’”

Angie made the point that test code monitors production code. In your CICD development, the state of the build depends on test acceptance. If you build sloppy test code, you run the risk of false positives and false negatives.

As your production code changes, your test code must change as well. The sloppier your test code, the more difficult time you will have in test maintenance. 

Writing test code with the same care as you write production gives you the best chance to keep your CICD pipeline in fast, consistent delivery. Alternatively, Angie said, if your test code stays a mess, you will have a tendency to avoid code maintenance. Avoiding maintenance will lead to untrustworthy builds. 

Writing UI Tests – Introduction

Next, Angie introduced the application she and Jessica were using for their coding demonstration. The app – a chat app, looks like this:

The welcome screen asks you to enter your username and click “Start Chatting” – the red button. Once you have done so, you’re in the app. Going forward, you enter text and click the “Send” button and it shows up on a chat screen along with your username. Other users can do the same thing.

With this as a starting point, Angie and Jessica began the process of test writing. 

Writing UI Tests – Coding Tests

Angie and Jessica were on a LiveShare of code, which looked like this:

From here, Angie started building her UI tests for the sign-in functionality. And, because she likes to code in Java, she coded in Java. 

All the objects she used were identified in the BaseTests class she inherited.

Her full code to sign-in looked like this:

public class ChattyBotTests extends BaseTests {
  private ChatPage chatPage:

  @Test
  public void newSession(){
     driver.get(appUrl);
     homePage.enterUsername("angie");
     chatPage = homePage.clickStartChatting();
     validateWindow();
  }

The test code gest the URL previously defined in the BaseTests class, fills in the username box with “angie”, and clicks the “Start Chatting” button. Finally, Angie added the validateWindow() method inherited from BaseTests, which uses Applitools visual testing to validate the new screen after the Start Chatting button has been clicked.

Next, Angie wrote the code to enter a message, click send message, and validate that the message was on the screen.

  @Test
  public void enterMessage(){
     chatPage.sendMessage("hello world");
     validateWindow():
 }

The inherited chatPage.sendMessage method both enters the text and clicks the Send Message button. validateWindow() again checks the screen using Applitools.

Are these usable as-is for CICD? Nope.

Coding Pre-Test Setup

If we want to run tests in parallel, these tests, as written, block parallel operation, since the enterMessage() depends on the newSession() being run previously.

So solve this, Angie creates a pre-test startSession() that runs before all tests. It includes the first three lines of newSession() which go to the app URL, enter “angie” as the username, and click the “Start Chatting” button. Next, Angie modifies her newSession() test so all it does is the validation.

  @Before
  public void startSession(){
     driver.get(appUrl);
     homePage.enterUsername("angie");
     chatPage = homePage.clickStartChatting();
  }

  @Test
  public void newSession(){
     validateWindow();
 }

With this @Before setup, Angie can create independent tests.

Adding Multi-User Test

Finally, Angie added a multi-user test. In this test, she assumed the @Before gest run, and her new test looked like this:

  @Test
  public void multiPersonChat(){

     //Angie sends a message
     chatPage.sendMessage(“hello world”);

     //Jessica sends a message
     WindowUtils.openNewTab(driver, appUrl);
     homePage.enterUsername("jessica");
     chatPage = homePage.clickStartChatting();
     chatPage.sendMessage("goodbye world");
     WindowUtils.switchToTab(driver, 1);
     validateWindow();
  }

Here, user “angie” sends the message “hello world”. Then, Angie codes the browser to:

• open a new tab for the app URL, 
• create a new chat session for “jessica”
• has “jessica” send the message “goodbye world”
• Switch back to the original tab
• Validate the window

Integrating UI Tests Into CICD

Now, it was Jessica’s turn to control the code. 

Before she got started coding, Jessica shared her screen from Visual Studio Code, to demonstrate the LiveShare feature of VS Code:

Angie and Jessica were working on the same file using LiveShare. LiveShare highlights Angie’s cursor on Jessica’s screen. 

When Angie selects a block of text, the text gets highlighted on Jessica’s screen.

This extension to Visual Studio Code makes it easy to collaborate on coding projects remotely. It’s available for download on the Visual Studio Code Marketplace. It’s great for pair programming when compared with remote screen share.

To begin the discussion of using these tests in CICD, Jessica started describing the environment for running the tests from a developer perspective versus a CICD perspective. A developer might imagine running locally, with IntelliJ or command line opening up browser windows. In contrast, CICD needs to run unattended.  So, we need to consider headless.

Jessica showed how she coded for different environments in which she might run her tests.

Her code explains that the environment gets defined by a variable called runWhere, which can equal one of three values:

• local – uses a ChromeDriver
• pipeline – uses a dedicated build server and sets the options –headless and –no-sandbox for ChromeDriver (note: for Windows you add the option “–disable-gui”)
• container – instructs the driver to be a remote web driver based on the selenium hub remote URL and passes the –headless and –no-sandbox chromeOptions

Testing Locally

First, Jessica needed to verify that the testa ran using the local settings.

Jessica set the RUNWHERE variable to ‘local’ using the command

export RUNWHERE=local

She had already exported other settings, such as her Applitools API Key, so she can use Applitools.

Since Jessica was already in her visual test folder, she run her standard maven command:

mvn -f visual_tests/pom.xml clean test

The tests ran as expected with no errors. The test opened up a local browser window and she showed the tests running.

Testing Pipeline

Next, Jessica set up to test her pipeline environment settings. 

She changed the RUNWHERE variable using the command:

export RUNWHERE=pipeline

Again, she executed the same maven tests

mvn -f visual_tests/pom.xml clean test

With the expectation that the tests would run as expected using her pipeline server, meaning that the tests run without opening a browser window on her local machine.

This is important because whatever CICD pipeline you use – Azure DevOps, Github Actions, Travis CI, or any traditional non-container-based CICD system – will want to use this headless interaction with the browser that keeps the GUI from opening up and possibly throwing an error.

Once these passed, Jessica moved on to testing with containers.

Testing Containers

Looking back, the container-based tests used a call to RemoteWebDriver, which in turns called selenium_hub:

Selenium_hub let Jessica spin up whatever browser she wanted.  To specify what she wanted, she used a docker-compose file, docker-compose.yaml:

These container-based approaches align with the current use of cloud-native pipelines for CICD. Jessica noted you can use Jenkins, Jenkins X for Kubernetes native, and CodeFresh, among others. Jessica decided to show CodeFresh. It’s a CICD pipeline dedicated to Kubernetes and microservices.  Every task runs in a container.

Selenium_Hub let Jessica choose to run tests on both a chorme_node and a firefox_node in her container setup.  

She simply needed to modify her RUNWHERE variable

export RUNWHERE=container

However, before running her tests, she needed to spin up her docker-compose on her local system. And, because selenium_hub wasn’t something that her system could identify by DNS at that moment (it was running on her local system), she ensured that the selenium_hub running locally would port forward onto her local system’s 127.0.0.1 connection. Once she made these changes, and changed the container definition to use 127.0.0.1:4444, she was ready to run her maven pom.xml file.

When the tests ran successfully, her local validation confirmed that her tests should run in her pipeline of choice.

Jessica pointed out that CICD really comes down to a collection of tasks you would run manually.

After that, Jessica said, we need to automate those tasks in a definition file. Typically, that’s Yaml, unless you really like pain and choose Groovy in Jenkins… (no judgement, she said).

Looking at Azure DevOps

Next, Jessica did a quick look into Azure DevOps.

Inside Azure DevOps, Jessica showed that she had a number of pipelines already written, and she chose the one she had set aside for the project. This pipeline already had three separate stages: 

• Build Stage
• Deploy to Dev
• Deploy to Prod

Opening up the build stage shows all the steps contained just within that stage in its 74 seconds of runtime:

Jessica pointed out that this little ChattyBot application is running on a large cluster in Azure. It’s running in Kubernetes, and it’s deployed with Helm.  The whole build stage includes:

• using JFrog to package up all the maven dependencies and run the maven build
• jfrog xray to make sure that the dependencies don’t result in security errors, 
• Creating a helm chart and packaging that,
• Sending Slack notifications

This is a pretty extensive pipeline. Jessica wondered how hard it would be to integrate Angie’s tests into an existing environment.

But, because of the work Jessica had done to make Angie’s tests ready for CICD, it was really easy to add those tests into the deploy workflow.

First, Jessica reviewed the Deploy to Dev stage.

Adding UI Tests in Your CICD Pipeline

Now, Jessica started doing the work to add Angie’s tests into her existing CICD pipeline. 

After the RUNWHERE=container tests finished successfully, Jessica went back into VS Code, where she started inspecting her azure-pipelines.yml file.

Jessica made it clear that she wanted to add the tests everywhere that it made sense prior to promoting code to production:

• Dev
• Test
• QA
• Canary

Jessica reinforced Angie’s earlier points – these UI tests were critical and needed to pass. So, in order to include them in her pipeline, she needed to add them in an order that makes sense.

In her Deploy to Dev pipeline, she added the following:

   - bash:
     # run check to see when $(hostname) is available
     attempt_counter=0
     max_attempts=5
     until $(curl --output /dev/null --silent --head --fail https://”$(hostname)”/); do
       if [ ${attempt_counter} -eq ${max_attempts} ]; then
         echo “Max attempts reached”
         exit 1
       fi

       printf “.”
       attempt_counter=$((attempt_counter+1))
       sleep 20
     done
   displayName: HTTP Check

This script checks to see if the url $hostname is available and gives up if not available after five tries after sleeping 20 seconds. Each try it displays a “.” to show it is working. And, the name “HTTP Check” shows what it is doing. 

Now, to add the tests, Jessica needed to capture the environment variable declarations and then run the maven commands.  And, as Jessica pointed out, this is where things can become challenging, especially when writing the tests from scratch, because people may not know the syntax.   

Editing the azure-pipelines.yml in Azure DevOps

Now, Jessica moved back from Visual Studio Code to Azure DevOps, where she could also edit an azure-pipelines.yml file directly in the browser.

And, here, on the right side of her screen (I captured it separately) are tasks she can add to her pipeline. The ability to add tasks makes this process really, really simple and eliminates a lot of the errors that can happen when you code by hand.

One of those tasks is an Applitools Build Task that she was able to add by installing an extension.

Just clicking on this Applitools Build Task adds it to the azure_pipelines.yml file.  

And, now Jessica wanted to add her maven build task – but instead of doing a bash script, she wanted to use the maven task in Azure DevOps. Finding the task and clicking on it shows all the options for the task.

The values are all defaults. Jessica changed the address for her pom.xml file to visual_tests/pom.xml (the file location for the test file), set her goal as ‘test’ and options as ‘clean test’. She checked everything else, and since it looked okay, she clicked the “Add” button.  The following code got added to her azure-pipelines.yml file.

  - task: Maven
     inputs: 
       mavenPomFile: ‘visual_tests/pom.xml’
       goals: 'test'
       options: 'clean test'
       publishJUnitResults: true
       testResultsFiles: '**/surefire-report/TEST-*.xml'
       javaHomeOption: 'JDKVersion'
       mavenVersionOption: 'Default'
       mavenAuthenticationFeed: false
       effectivePomSkip: false
       sonarQubeRunAnalysis: false

Going Back To The Test Code

Jessica copied the Applitools Built Task and Maven task code file back into the azure-pipelines.yml file she was already editing in Visual Studio Code.

Then, she added the environment variables needed to run the tests.  This included the Applitools API Key, which is a secret value from Applitools. In this case, Jessica defined this variable in Azure DevOps and could call it by the variable name.

Beyond the Applitools API Key, Jessica also set the RUNWHERE environment variable to ‘pipeline’ and the TEST_START_PAGE environment variable to the $hostname – same as used elsewhere in her code. All this made her tests dynamic.  

The added code reads:

     env: 
       APPLITOOLSAPIKEY: $APPLITOOLS_API_KEY
       RUNWHERE: pipeline
       TEST_START_PAGE: https://($hostname)/

So, now, the tests are ready to commit.

One thing Jessica noted is that LiveShare automatically adds the co-author’s id to the commit whenever two people have jointly worked on code. It’s a cool feature of LiveShare.

Verifying That UI Tests Work In CICD

So, now that the pipeline code had been added, Jessica wanted to demonstrate that the visual validation with Applitools worked as expected and found visual differences.   

Jessica modified the ChattyBot application so that, instead of reading:

“Hello, DevOps Days Madrid 2020!!!”

it read:

“Hello, awesome webinar attendees!”

She saved the change, double-checked the test code, saw that everything looked right, and pushed the commit.

This kicked off a new build in Azure DevOps. Jessica showed the build underway. She said that, with the visual difference, we expect the Deploy to Dev pipeline to fail. 

Since we had time to wait, she showed what happened on an earlier build that she had done just before the webinar. During that build, the Deploy to Dev passed. She was able to show how Azure DevOps seamlessly linked the Applitools dashboard – and, assuming you were logged in, you would see the dashboard screen just by clicking on the Applitools tab.

Here, the green boxes on the Status column show that the tests passed.

Jessica drilled into the enterMessage test to show how the baseline and the new checkpoint compared (even though the comparison passed), just to show the Applitools UI.  

As Jessica said, were any part of this test to be visually different due to color, sizing, text, or any other visual artifact, she could select the region and give it a thumbs-up to approve it as a change (and cause the test to pass), or give it a thumbs-down and inform the dev team of the unexpected difference.

And, she has all this information from within her Azure DevOps build.

What If I Don’t Use Azure DevOps?

Jessica said she gets this question all the time, because not everyone uses AzureDevOps.  

You could be using Azure DevOps, TeamCity CI, Octopus Deploy, Jenkins – it doesn’t matter. You’re still going to be organizing tasks that make sense.  You will need to run an HTTP check to make sure your site is up and running.  You will need to make sure you have access to your environment variables. And, then, finally, you will need to run your maven command-line test.  

Jessica jumped into Github Actions, where she had an existing pipeline, and she showed that her deploy step looked identical.

It had an http check, an Applitools Build Task, and a call for Visual Testing. The only difference was that the Applitools Build Task included several lines of bash to export Applitools environment variables.

The one extra step she added, just as a sanity check, was to set the JDK version.

And, while she was in Github Actions, she referred back to the container scenario. She noted the challenges with spinning up Docker Compose and services.  For this reason, when looking at container tests, she pointed to CodeFresh, which is Kubernetes-native.

Inside her CodeFresh pipelines, everything runs in a container.

As she pointed out, by running on CodeFresh, she didn’t need a huge server to handle everything. Each container handled just what it needed to handle. Spinning up Docker Compose just requires docker. She needed just jFrog for her Artifactory image. Helm lint – again, just what she needed.

The image above shows the pipelines before adding the visual tests. The below image shows the Deploy Dev pipeline with the same three additions.

There’s the HTTP check, the Applitools Build Task, and Running Visual Tests.

The only difference really is that the visual tests ran alongside services that were spinning up alongside the test.

This is really easy to do in your codefresh.yml file, and the syntax looks a lot like Docker Compose.

Seeing the Visual Failure

Back in Azure DevOps, Jessica checked in on her Deploy to Dev step.  She already knew there was a problem from her Slack notifications.  

The error report showed that the visual tests all failed.

Clicking on the Applitools tab, she saw the following.

All three tests showed as unresolved.  Clicking in to the multiPersonChat test, Jessica saw this:

Sure enough, the text change from “Hello, DevOps Days Madrid 2020!!!” to “Hello, awesome webinar attendees!” caused a difference. We totally expected this difference, and we would find that this difference had also shown up in the other tests.

The change may not have been a behavioral change expected in your tests, so you may or may not have thought to test for the “Hello…” text or check for its modification. Applitools makes it easy to capture any visual difference.

Jessica didn’t go through this, but one feature in Applitools is the ability to use Auto Maintenance. With Auto Maintenance, if Jessica had approved the change on this first page, she could automatically approve identical changes on other pages. So, if this was an intended change, it would go from “Unresolved” to “Passed” on all the pages where the change had been observed.

Summing Up

Jessica handed back presentation to Angie, who shared Jessica’s link for code from the webinar:

All the code from Angie and Jessica’s demo can be downloaded from:

https://aka.ms/jldeen/applitools-webinar

Happy Testing!

For More Information

• Read Ask 288 Of Your Peers About Visual AI
• Read How I ran 100 UI tests in just 20 seconds
• Take Angie Jones’s course on Removing Visual Blind Spots
• Sign up for Test Automation University and start taking classes
• Take Raja Rao’s Course on Modern Functional Testing
• Sign up for a free Applitools account
• Request an Applitools demo

The post UI Tests In CICD – Webinar Review appeared first on Automated Visual Testing | Applitools.

If you run continuous deployment today, you need high-performance testing. You know the key takeaway shared by our guest presenter, Priyanka Halder: test speed matters.

Priyanka Halder presented her approach to achieving success in a hyper-growth company through her webinar for Applitools in January 2020. The title of her speech sums up her experience at GoodRx:

“High-Performance Testing: Acing Automation In Hyper-Growth Environments.”

Hyper-growth environments focus on speed and agility. Priyanka focuses on the approach that lets GoodRx not only develop but also test features and releases while growing at an exponential rate.

About Priyanka

Priyanka Halder is head of quality at GoodRx, a startup focused on finding all the providers of a given medication for a patient – including non-brand substitutes – and helping over 10 million Americans find the best prices for those medications.  Priyanka joined in 2018 as head of quality engineering – with a staff of just one quality engineer. She has since grown the team 1200% and grown her team’s capabilities to deliver test speed, test coverage, and product reliability. As she explains, past experience drives current success.

Priyanka’s career includes over a dozen years of test experience at companies ranging from startups to billion-dollar companies. She has extensive QA experience in managing large teams and deploying innovative technologies and processes, such as visual validation, test stabilization pipelines, and CICD. Priyanka also speaks regularly at testing and technology conferences. She accepted invitations to give variations of this particular talk eight times in 2019.

One interesting note: she says she would lik to prove to the world that 100% bug-free software does not exist.

Start With The Right Foundation

Priyanka, as a mother, knows the value of stories. She sees the story of the Three Little Pigs as instructive for anyone trying to build a successful test solution in a hyper-growth environment. Everyone knows the story: three pigs each build their own home to protect themselves from a wolf. The first little pig builds a straw house in a couple of hours. The second little pig builds a home from wood in a day. The third little pig builds a solid infrastructure of brick and mortar – and that took a number of days. When the wolf comes to eat the pigs, he can blow down the straw house and the wood house, but the solid house saves the pigs inside.

Priyanka shares from her own experience. – She encounters many wolves in a hyper-growth environment. The only safeguard comes from building a strong foundation. Priyanka describes a hyper-growth environment and how high-performance testing works.  She describes the technology and team needed for high-performance testing. And, she describes what she delivered (and continues to deliver) at GoodRx.

Define High-Performance Testing

So, what is high-performance testing?

Fundamentally, high-performance testing maximizes quality in a hyper-growth startup. To succeed, she says, you must embrace the ever-changing startup mentality, be one step ahead, and constantly provide high-quality output without being burned out.

Agile startups share many common characteristics:

• Chaotic – you need to be comfortable with change
• Less time – all hands on deck all the time for all the issues
• Less resources – you have to build a team where veterans are mentors and not enemies
• Market pressure – teams need to understand and assess risk
• Reward – do it right and get some clear benefits and perks

If you do it right, it can lead to satisfaction. If you do it wrong, it leads to burnout. So – how do you do it right?

Why High-Performance Testing?

Leveraging data collected by another company Priyanka showed how the technology for app businesses changed drastically over the past decade. These differences include:

• Scope – instead of running a dedicated app, or on a single browser, today’s apps run on multiple platforms (web app and mobile)
• Frequency – we release apps on demand (not annually, quarterly, monthly or daily)
• Process – we have gone from waterfall to continuous delivery
• Framework – we used to use singe-stack on premise software – today we are using open source, best of breed, cloud based solutions for developing and delivering.

The assumptions of “test last” that may have worked a decade back can’t work anymore. So, we need a new paradigm.

How To Achieve High-Performance Testing

Priyanka talked about her own experience. Among other things, teams need to know that they will fail early as they try to meet the demands of a hyper-growth environment. Her approach, based on her own experiences, is to ask questions:

• Does the team appreciate that failures can happen?
• Does the team have inconsistencies? Do they have unclear requirements? Set impossible deadlines? Use waterfall while claiming to be agile? Note those down.

Once you know the existing situation, you can start to resolve contradictions and issues. For example, you can use a mind map to visualize the situation. You can divide issues and focus on short term work (feature team for testing) vs. long term work (framework team). Another important goal – figure out how to find bugs early (aka Shift Left). Understand which tools are in place and which you might need. Know where you stand today vis-a-vis industry standards for release throughput and quality. Lastly, know the strength of your team today for building an automation framework, and get AI and ML support to gain efficiencies.

Building a Team

Next, Priyanka spoke about what you need to build a team for high-performance testing.

In the past, we used to have a service team. They were the QA team and had their own identity. Today, we have true agile teams, with integrated pods where quality engineers are the resource for their group and integrate into the entire development and delivery process.

So, in part you need skills. You need engineers who know test approaches that can help their team create high-quality products. Some need to be familiar with behavior-driven design or test-driven design. Some need to know the automation tools you have chosen to use. And, some need to be thinking about design-for-testability.

One huge part of test automation involves framework. You need a skill set familiar with building code that self-identifies element locators, builds hooks for automation controls, and ensures consistency between builds for automation repeatability.

Beyond skills, you need individuals with confidence and flexibility. They need to meld well with the other teams. In a truly agile group, team members distribute themselves through the product teams as test resources. While they may connect to the main quality engineering team, they still must be able to function as part of their own pod.

Test Automation

Priyanka asserts that good automation makes high-performance testing possible.

In days gone by, you might have bought tools from a single vendor. Today, open source solutions provide a rich source for automation solutions. Open source generally has lower maintenance costs, generally lets you ship faster, and expands more easily.

Open source tools come with communities of users who document best practices for using those tools. You might even learn best-practice processes for integrating with other tools. The communities give you valuable lessons so you can learn without having to fail (or learn from the failures of others).

Priyanka describes aspects of software deployment processes that you can automate.  Among the features and capabilities you can automate:

• Assertions on Action
• Initialization and Cleanup
• Data Modeling/Mocking
• Configuration
• Safe Modeling Abstractions
• Wrappers and Helpers
• API Usage
• Future-ready Features
• Local and Cloud Setups
• Speed
• Debugging Features
• Cross Browser
• Simulators/Emulators/Real Devices
• Built-in reporting or easy to plug in

Industry Standards

You can measure all sorts of values from testing. Quality, of course. But what else? What are the standards these days? Who knows what are typical test times for test automation?

Priyanka shares data from Sauce Labs about standards.  Sauce surveyed a number companies and discussed benchmark settings for four categories: test quality; test run time; test platform coverage; and test concurrency. The technical leaders at these companies set some benchmarks they thought aligned with best-in-class industry standards.

In detail:

• Quality – pass at least 90% of all tests run
• Run Time – average of all tests run two minutes or less
• Platform Coverage – tests cover five critical platforms on average
• Concurrency – at peak usage, tests utilize at least 75% of available capacity

Next, Priyanka shared the data Sauce collected from the same companies about how they fared against the average benchmarks discussed.

• Quality – 18% of the companies achieved 90% pass rate
• Run time – 36% achieved the 2 minute or less average
• Platform coverage – 63% reached the five platform overage
• Concurrency – 71% achieved the 75% utilization mark
• However, only 6.2% of the companies achieved the mark on all four.

Test speed became a noticeable issue. While 36% ran on average in two minutes or faster, a large number of companies exceeded five minutes – more than double.

Investigating Benchmarks

These benchmarks are fascinating – especially run time – because test speed is key to faster overall delivery. The longer you have to wait for testing to finish, the slower your dev release cycle times.

Sadly, lots of companies think they’re acing automation, but so few are meeting key benchmarks. Just having automation doesn’t help. It’s important to use automation that helps meet these key benchmarks.

Another area worth investigating involves platform coverage. While Chrome remains everyone’s favorite browser, not everyone is on Chrome.  Perhaps 2/3 of users run Chrome, but Firefox, Safari, Edge and others still command attention. More importantly, lots of companies want to run mobile, but only 8.1% of company tests run on mobile. Almost 92% of companies run desktop tests and then resize their windows for the mobile device.  Of the mobile tests, only 8.9% run iOS native apps and 13.2% run Android native apps. There’s a gap at a lot of companies.

GoodRx Strategies

Priyanka dove into the capabilities that allow GoodRx to solve the high- performance testing issues.

Test In Production

The first capabilities GoodRx uses a Shift Right approach that moves testing into the realm of production.

Production testing? Yup – but it’s not spray-and-pray. GoodRx’s approach includes the following:

• Feature Flag – Test in production. Ship fast, test with real data.
• Traffic Allocation – gradually introduce new features and empower targeted users with data. Hugely important for finding corner cases without impacting the entire customer base.
• Dog Fooding – use a CDN like Fastly to deploy, route internal users to new features.

The net reduce – this reduces overhead, lets the app get tested with real data test sets, and identify issues without impacting the entire customer base. So, the big release becomes a set of small releases on a common code base, tested by different people to ensure that the bulk of your customer base doesn’t get a rude awakening.

AI/ML

Next, Priyanka talked about GoodRx uses AI/ML tools to augment her team. These tools make her team more productive – allowing her to meet the quality needs of the high-performance environment.

First, Priyanka discussed automated visual regression – using AI/ML to automate the validation of rendered pages. Here, she talked about using Applitools – as she says, the acknowledged leader in the field. Priyanka talked about how GoodRx uses Applitools.

At GoodRx, there may be one page used for a transaction. But, GoodRx supports hundreds of drugs in detail, and a user can dive into those pages that describe the indications and cautions about individual medications.  To ensure that those pages remain consistent, GoodRx validates these pages using Applitools. Trying to validate these pages manually would take six hours. Applitools validates these pages in minutes and allows GoodRx to release multiple times a day.

To show this, Priyanka used an example of visual differences. She showed a kids cartoon with visual differences. Then she showed what happens if you do a normal image comparison – pixel-based comparison.

A bit-wise comparison will fail too frequently.  Using the Applitools AI system, they can set up Applitools to look at the images that have already been approved and quickly validate the pages being tested.

Applitools can complete a full visual regression in less than 12 minutes to run 350 test cases, which runs 2,500 checks.  Manually, it takes six hours.

Priyanka showed the kinds of real-world bugs that Applitools uncovered. One – a screenshot from her own site GoodRx. A second from amazon.com, and a third from macys.com. She showed examples with corrupt display – and ones that Selenium alone could not catch.

ReportPortal.io

Next, Priyanka moved on to ReportPortal.io. As she says, when you ace automation, you need to know where you stand. You need to build trust around your automation platform by showing how it is behaving. All your data – test times, bugs discovered, etc. reportportal.io shows how tests are running at different times of the day.  Another display shows flakiest tests and longest-running tests to help the team release seamlessly and improve their statistics.

Any failed test case in reportportal.io can link the test results log directly into the reportportal.io user interface.

GoodRx uses behavior-driven design (BDD), and their BDD approach lets them describe the behavior they want for a given feature – how it should behave in good and bad cases – and ensure that those cases get covered.

High-Performance Testing – The Burn Out

Priyanka made it clear that high-performance environments take a toll on people. Everywhere.

She showed a slide referencing a blog by Atlassian talking about work burnout symptoms – and prevention. From her perspective, the symptoms of workplace stress include:

• Being cynical or critical at work
• Dragging yourself to work and having trouble getting started
• Irritable or impassion, lack energy, hard to concentrate, headache
• Lack of satisfaction from achievement
• Use food, drugs or alcohol to feel better or simply not to feel

So, what should a good team lead do when she notices signs of burnout? Remind people to take steps to prevent burnout. These include:

• Avoid unachievable deadlines. Don’t take on too much work. Estimate, add buffer, add resource.
• Do what gives you energy – avoid what drains you
• Manage digital distraction – the grass will always be greener on the other side
• Do something outside your work – Engage in activities that bring you joy
• Say No too many projects – gauge your bandwidth and communicate
• Make self-care a priority – meditation/yoga/massage
• Have a strong support system – talk to you family, friends, seek help
• Unplugging for short periods helps immensely

The point here is that hyper-growth environments can take a toll on everyone – employees, managers. Unrealistic demands can permeate the organization. Use care to make sure that this doesn’t happen to you or your team.

GoodRx Case Study

Why not look at Priyanka’s direct experience at GoodRx? Her employer, GoodRx, provides prices transparency for drugs. GoodRx lets individuals search for drugs they might need or use for various conditions. Once an individual selects a drug, GoodRx lets the individual see the prices for that drug in various locations to find the best price for that drug.

The main customers are people who don’t have insurance or have high-deductible insurance. In some cases, GoodRx offers coupons to keep the prices low.  GoodRx also provides GoodRx Care – a telemedicine consultation system – to help answer patient questions about drugs. Rather than see a doctor, GoodRx Care costs anywhere between $5 and $20 for a consultation.

Because the GoodRx web application provides high value for its customers, often with high demand, the app must maintain proper function, high performance, and high availability.

Set Goals

The QA goals Priyanka designed needed to meet the demands of this application. Her goals included:

• Distributed QA team 24/7 QA support
• Dedicated SDET Team who specializes in test
• A robust framework that will make any POC super simple (plug and play)
• Test stabilization pipeline using Travis
• 100% automation support to reduce regression time 90%

Build a Team

As a result, Priyanka needed to hire a team that could address these goals. She showed the profile she developed on LinkedIn to find people that met her criteria – dev-literate, test-literate engineers who could work together as a team and function successfully. More emphasis on test automation and coding abilities rose to the top.

Build a Tech Stack

Next, Priyanka and her team invested in tech stack:

• Python and Selenium WebDriver
• Behave for BDD
• Browserstack for a cloud runner
• Applitools for visual regression
• Jenkins/Travis and Google Drone for CI
• Jira, TestRail for documentation

CICD success criteria requirements came up with four issues:

• Speed and parallelization
• BDD for easy debug and read
• Cross-browser cross-device coverage in CICD
• Visual validation

Set QA expectations for CI/CD testing

Finally, Priyanka and her team had to set expectations for testing.  How often would they test? How often would they build?

The QA for CI/CD means that test and build become asynchronous. Regardless of the build state,

• Hourly; QA runs 73 tests hourly against the latest build to sanity check the site.
• On Build: Any new build runs 6 cross-browser and makes sure all critical business paths get covered.
• Nightly 300 test regression tests on top of other tests.

Some of these were starting points, but most got refined over time.

Priyanka’s GoodRx Quality Timeline

Next, Priyanka talked about how her team grew from the time she joined until now.

She started in June 2018. At that point, GoodRx had one QA engineer.

• In her first quarter, she added a QA Manager, QA Analyst, and a Senior SDET. They added offshore reprocessing to support releases.
• By October 2018 they had fully automated P0/P1 tests. Her team had added Spinnaker pipeline integration. They were running cross-browser testing with real mobile device tests.
• By December 2018 she added two more QA Analysts and 1 more SDET.  Her team’s tests fully covered regression and edge cases.
• And, she pressed on. In early 2019, they had built automation-driven releases. They had added Auth0 support – her team was hyper-productive.
• Then, she discovered her team had started to burnout.  Two of her engineers quit. This was an eye-opening time for Priyanka. Her lessons about burnout came from this period. She learned how to manage her team through this difficult period.

By August 2019 she had the team back on an even keel and had hired three QA engineers and one more SDET.

And, in November 2019 they achieved 100% mobile app automation support.

GoodRx Framework for High-Performance Testing

Finally, Priyanka gave a peek into the GoodRx framework, which helps her team build and maintain test automation.

The browser base class provides access for test automation. Using the browser base class eliminates the need to use Selenium embed click.

The page class simplifies the web element location. The page class structure assigns a unique XPath to each web element. Automation benefits by having clean XPath elements for automation purposes.

The element wrapper class allows for behaviors like lazy loading.  Instead of having to program exceptions into the test code, the element wrapper class standardizes interaction between the browser under test and the test infrastructure.

Finally, for every third-party application or tool that integrates using an SDK, like Applitools GoodRx deploys an SDK Wrapper. As one of her SDET team figured, the wrapper ensures that an SDK change from a third party can mess up your test behavior. Using a wrapper is a good practice for handling situations when the service you use encounters something unexpected.

The framework results in a more stable test infrastructure that can rapidly change to meet the growth and change demands of GoodRx.

Conclusions

Hyper-growth companies put demands on their quality engineers to achieve quickly. Test speed matters, but it cannot be achieved consistently without investment. Just as Priyanka started with the story of the Three Little Pigs, she made clear that success requires investment in automation, people, AI/ML, and framework.

To watch the entire webinar:

For More Information

• Whole Team Approach to Continous Testing (Blog)
• Whole Team Approach to Continous Testing (TAU Course)
• How Visual UI Testing can speed up DevOps flow (Blog)
• Integrate BitBucket into your Visual Testing (Blog)
• Get started with Visual Validation (Blog)
• Sign up for a free Applitools Account
• Request an Applitools demo

The post How To Ace High-Performance Test for CI/CD appeared first on Automated Visual Testing | Applitools.

Watch Joe Colantonio interview Greg Sypolt — Director of Quality Engineering @ Gannett | USA Today — and learn how he built his Dev/Test team for success – including: hiring test engineers, organizational culture, must-have tools, work processes, and best practices. 

Improving coverage, quality, and teamwork — which lead to faster, better releases — requires a seamless merger of leadership skills, engineering savvy, advanced best-in-market tools, and cross-team collaboration and communication. But although most organizations claim they’re committed to agile development and digital transformation — including fully automated CI-CD processes — this is not a simple task: 75% of companies that have started this journey testified to failure in achieving those goals.

So what is the secret of success of the 25% of companies that have triumphed with implementing this change?

Greg Sypolt — Director of Quality Engineering @ Gannett | USA Today Network — will share the steps he took on his successful quest to conquer a fully automated CI-CD release pipeline, while supporting dozens of different apps, across a myriad of digital platforms covering thousands of end-user engagement points.

In this live interview hosted by Joe Colantonio — founder of the Guild Conferences and TestTalks podcast — you will discover what is Greg’s secret recipe for building a winning Dev/Test team that delivers quality software products – faster.

Watch this live interview and learn:

• Hiring for success: building a winning Dev/Test team
• Establishing and maintaining a culture of teamwork and effective communication
• Improving test coverage with more accuracy and less code
• Accelerating release cycles, while improving quality but without going over-budget
• Tools of the trade: what is in Greg’s tech stack that he can’t — and won’t — do without

Full Webinar Recording:

Greg is hiring for his team!

Have you got what it takes? Check out the job requirements and apply here

Additional Resources

Here is some recommended reading to learn more:

From Greg Sypolt

• Learn more about Simulato and model-based testing go to our website – https://gannettdigital.github.io/simulato/
• Learn more about CI/CD, check out my keynote next month at Agile + DevOps West conference in Las Vegas: The Evolution of a Continuous Integration Pipeline
• These 2 blog posts might be a good reference – rethinking the role ‘journey’ in software testing + tips for achieving continuous improvement in software testing
• You can find all my blog posts and conference recordings @ http://www.qualityelement.com/blog/category/greg-sypolt AND http://www.qualityelement.com/gregsypolt-982108.html

From Joe Colantonio

Special Offer — 30% Off on Secure Guild Conf Tickets!

From Applitools’ Content Editors

1. Webinar: DevOps & Quality in The Era Of CI-CD: An Inside Look At How Microsoft Does It — with Abel Wong of Microsoft Azure DevOps
2. How to Run 372 Cross Browser Tests In Under 3 Minutes — post by Jonah Stiennon
3. How Visual Regression Testing Can Help You Deliver Better Apps — post by Jay Phelps
4. Challenges of Testing Responsive Design – and Possible Solutions — post by Justin Rohrman
5. Webinar: Creating a Flawless User Experience, End-to-End, Functional to Visual — Practical Hands-on Session w. Greg Bahmutov (Cypress.io) and Gil Tayar (Applitools)
6. How to Do Visual Regression Testing with Selenium — by Dave Haeffner
7. Release Apps with Flawless UI: Open your Free Applitools Account, and start visual testing today.
8. Improve your skills and build your resume for Success — with Test Automation University! The most-talked-about test automation initiative of 2019: online education platform led by Angie Jones, offering free test automation courses by industry leaders. Enroll, and start showing off your test automation certificates and badges!

— HAPPY TESTING —

The post Quality, Coverage, and Teamwork: The Holy Trinity Of USA Today’s Winning Dev/Test Team appeared first on Automated Visual Testing | Applitools.

I regularly use Jenkins CI and GitHub to help our customers integrate Applitools into their continuous integration workflow. Even though I’m pretty familiar with both tools, there’s one thing that drives me nuts.

Here’s the problem:

I can’t see Jenkins build status in my GitHub repo pull request. Below is a screenshot of a typical GitHub pull request. As you can see, there’s no indication of my Jenkins build status.

This forces me to leave GitHub and go to Jenkins every time I merge, just so I can check the build status.

Doing this once is no big deal. But after doing this for a hundred pull requests, I feel like pulling my hair out. (And that would just look weird.)

This is especially frustrating since other continuous integration servers, like Travis and Circle CI, make this easy.

Since Jenkins is the third most popular continuous integration tool used with GitHub (after Travis and Circle CI), you’d think this issue would have been solved.

But when I started looking online for a way to solve this issue. I found some good resources, but not one document that described the process from beginning to end with all the configurations needed.

So, to spare you the pain I’ve gone through trying to make this work, I wrote this end-to-end tutorial on how to update Jenkins build status in the GitHub pull request page.

So let’s dive in!

The Big Picture

To get a better understanding of the interaction between those tools and the problem they present, see the following diagram. The red marks are the parts that are missing and this post is attempting to fix.

Get the basics in place

To follow the steps below, you just need a couple of things:

1. A GitHub account.
2. A Jenkins server up and running on a static URL.

What you will do in this tutorial:

1. Setup GitHub
   1. Install Jenkins plugin
   2. Generate a GitHub access token
2. Setup Jenkins
   1. Install Hudson Post Build Task plugin
   2. Call the GitHub API

   3) Bask in the awesomeness of your technical prowess

Setup GitHub

So, how to get GitHub setup with Jenkins? These steps explain.

What you’ll do here is install the Jenkins plugin on your GitHub repository in order to trigger a build in Jenkins when you commit a change in GitHub. This way, Jenkins can run tests to make sure your code is working as expected.

On your GitHub account, go to your repository and configure a webhook to the Jenkins server:

1. In your browser, go to your GitHub account. The URL should be something like www.github.com/your_username
2. Go into one of your repositories
3. Click on Settings, on the upper right part of the page
4. Click on Webhooks
5. Click on Add Webhook
6. Set the payload url to <http://your_jenkins_server_url>/github-webhook/
7. Set Content type to application/json
8. Select the event you would like the webhook to be triggered by (I set my webhook to be triggered by “Just the push event”.
9. Click on Add webhook

Here’s what those steps look like visually:

So, how to get GitHub setup with Jenkins? These steps explain.

What you’ll do here is install the Jenkins plugin on your GitHub repository in order to trigger a build in Jenkins when you commit a change in GitHub. This way, Jenkins can run tests to make sure your code is working as expected.

On your GitHub account, go to your repository and install the Jenkins plugin:

1. In your browser, go to your GitHub account. The URL should be something like www.github.com/your_username
2. Go into one of your repositories
3. Click on Settings, on the upper right part of the page
4. Click on Integrations and services
5. Click on Add service
6. Search for Jenkins
7. Add your Jenkins static URL or IP.
8. Click on Add service

Here’s what those steps look like visually. 

Generate a GitHub access token

You will need to generate a GitHub access token for security purposes. This token can be used by an application to use an API, here the application is Jenkins and the API is the GitHub API to update the Jenkins build status in the pull request.

 Ok, now you’re on to some non-obvious items.

1. Go to your GitHub account.
2. Under your profile, open the drop-down menu and click on Settings.

3. Go to Developer settings on the left side menu, all the way on the bottom:

4. Click on Personal access tokens:

5. On the top right corner of the screen click on the Generate new token button.

6. Insert your password.

7. Under repo check the repo:status checkbox.

9. Click on Generate token at the bottom of the page.

10. Make sure to save your token somewhere so you can use it later, such as in the curl commands above.

Jenkins setup

So, how to get Jenkins working with GitHub? Let’s show you how.

What you’ll do here is set up your Jenkins project to run your automation tests and call the GitHub API to update the status of the commit with success or failure.

Install GitHub plugin:

1. In your browser, go to your Jenkins URL
2. Click on Manage Jenkins
3. Click on Manage Plugins
4. Go to Available tab and search for GitHub integration plugin
5. Check the checkbox on the left and click on download and install after restart

Setup GitHub configuration:

1. Go to your GitHub project
2. Click on Configure
3. On the top bar, click on Source code management
4. Check the Git  checkbox
5. Paste the repository URL that you would like to integrate with under repository URL.
6. Set the Branch to integrate with under Branch specified.

Now that you have GitHub setup, do the following:

1. In your browser, go to your Jenkins URL
2. Click on Manage Jenkins
3. Click on Manage Plugins
4. Click on Install Hudson Post Build Task

Here’s what you’ll see when you run each of those commands. First, the Manage Jenkins command, in the menu bar on the left side:

Next, the Manage Plugins command.

And finally, the Install Hudson build task command.

(In case you’re wondering why you’re installing a Hudson plugin and not a Jenkins plugin, it’s because of this interesting bit of CI historical trivia.)

Install Hudson Post Build Task plugin

The next step is to go into your Jenkins project post build actions, and add two post build tasks. One task for success, and another for failure. You will need those actions to distinguish between a successful build and a build that failed.

• The plugin name could also be called “post build task”

Define those build actions as follows:

What you are doing is looking for the words “SUCCESS” or “FAILURE” in the Jenkins log. When you find “SUCCESS”, you run a script that uses the curl command to call the GitHub webhook API to set the build status of the commit that started the build in GitHub pull request. The commit identifier, the SHA, is accepted by Jenkins within an environment variable – $GIT_COMMIT. Similarly, if you find “FAILURE” in the Jenkins log, you use curl to let GitHub know that the Jenkins build failed.

Either way, the status will be updated in the pull request. And, if you run Applitools tests, the Applitools status will also be updated.

Call the GitHub API

For log entries containing “SUCCESS”, here’s the post-build command that you want Jenkins to run:

View the code on Gist.

And for log entries containing “FAILURE”, here’s the post-build command that you want Jenkins to run::

View the code on Gist.

Notice that you will need to replace the following with your relevant info:

• <GitHubUserName> – from GitHub
• <REPO_NAME> – from GitHub
• <YOUR_GITHUB_TOKEN>
• <YOUR_JENKINS_URL>
• <JenkinsProjectName> – your Jenkins project name

Let’s dig into these post-build commands, using the curl man pages.

As you can see, these are having curl request a URL from api.github.com. This URL has your GitHub username, repository name, and access token in it.

You’re sending a JSON snippet in the header of the HTTP request, as specified by the -H option. And you’re sending an HTTP POST request, as indicated by the -X flag. And the data that post command is sending, indicated by -d, is a bit of JSON with either success or failure, your Jenkins URL, and your Jenkins project name.

(That’s some pretty complex curl you got going!)

Once you get it setup be sure to give yourself a pat on the back. Or a nice Belgian waffle with whipped cream and strawberries. (I know which one I’d pick!)

So, where do you get all this information?

To get this info, you’ll do the following:

• GitHub Username – This should be your GitHub username. Hopefully this is obvious.
• GitHub Repository Name – You can find the name of any particular GitHub repo at the top of its page. In the example below, the repo name is Applitools-GitHubIntegration.

• Jenkins URL – This should be the base URL of your Jenkins server. This is simply the URL you put into your browser to access Jenkins, without anything to the right of the domain name. For example, if you have a Jenkins server setup on Amazon EC2, it might look something like this:

http://ec2-53-70-132-187.compute-1.amazonaws.com/

In order for GitHub to find your Jenkins server, you should plug in your fully qualified domain name.

• Jenkins Project Name – which you can see in the Jenkins UI as shown below.

Conclusion

Now that you’ve configured both Jenkins and GitHub, the commit status will be shown in the GitHub pull request page when you merge a PR:

As you can see, the Jenkins build status is now appearing in the pull request page. 

Pretty nice, right?

The post How to update Jenkins build status in GitHub pull requests [Step-by-Step Tutorial] appeared first on Automated Visual Testing | Applitools.

A few weeks back, Applitools made the trip just up the road from our headquarters to take part in DevOps World | Jenkins World San Francisco. The event is an international event that provides opportunities to learn, explore, network and help shape the future of DevOps and CI with Jenkins. It’s always an exciting and energizing event and this year proved no different as we were able to network and engage with leading minds in the world of software development.

While we were there, James Lamberti, our Chief Marketing Officer, had the honor of sitting down with Jayne Groll, CEO of the DevOps Institute, to talk shop and explore some of the ways Applitools has changed how organizations are able to use visual testing to deliver better UI/UX for their customers. Here are some of the highlights of that conversation. 

Understanding Applitools and How It Fits into Digital Transformations

Jayne’s first question led James to discuss the basics of what the Applitools Visual AI tool is – and how it aligned perfectly with the theme of the event.

“Applitools exists to make sure the user interface is visually perfect on every device, operating system, and screen, whether mobile and desktop,” said Lamberti. “The theme [of DevOps World] this year is transform. And essentially, the issue or the problem that we solve for pretty much everyone in this room is digital transformation.”

While “digital transformation” at times is viewed as an industry buzzword, it is inarguable that businesses of all sizes and verticals are using digital as a means of fundamentally transforming their businesses, in part by building applications to streamline customer interactions. So when it comes to the user interface a customer-facing app, there is truly no brand spared from the impact of what a broken UI can do.

Bad UI Can Keep Your Business Grounded

A real-world example James highlighted how a broken UI can fundamentally affect your business, and its bottom line, was his recent experience trying to book a flight via his mobile device. When it was time to finalize the transaction, the checkout button was unclickable – ending the purchase journey right then and there. As James pointed out, this sort of break in the user experience will cause your customer to switch on a dime to another company – and this is exactly the type of issue that can be prevented by partnering with Applitools.

Exciting Developments on the Horizon 

Finally, Jayne and James touched on some of the exciting news coming out of Applitools – namely our newest product, the world’s first UI version control system.

“Essentially, we have integrated with GitHub and, for the first time ever, you can actually tie your code commits into GitHub directly to changs to the UI. This means that, for the first time, you can correlate code changes to UI changes, and tracking the history of how your UI evolves over time,” said James. “And so now, if there’s a UI issue, you can tie it back to what code commit caused that issue.”

For instance, if you see that an application stopped meeting its business goals a month ago — say, transactions declined, or user engagement slipped — you can look at the visual changes that were made to your app’s UI around that time, in order to discern if the application might be to blame. Furthermore, you know what changes to undo in your code to get your UI back to its previous state.

It sounds simple concept — and it is. What’s shocking, to us anyway, is that no one has built this fundamental capability for DevOps teams. It’s a crucial piece of application management – and one we are excited to bring to our customers. We think having the ability to have the visual UI tied to its code is transformative for how we approach some of the challenges around digital transformation.

Watch the full interview discussion with Jayne Groll at Jenkins World:

If you weren’t able to connect with us at DevOps World | Jenkins World, reach out or sign up for a free Applitools account.

=====

FULL TRANSCRIPT:

Jayne: Hi everyone. I’m Jayne Groll, CEO of the DevOps Institute here at DevOps World | Jenkins World in San Francisco. And I’m really excited to be joined by James Lamberti of the Applitools. Welcome.

James: Thank you. Great for you to have us. I appreciate you doing what you’re doing here. It’s exciting.

Jayne: It’s really exciting, right? I don’t know if anybody can hear the buzz in the background but the energy is really palpable.

James: It definitely is.

Jayne: Yeah, interesting. So James, tell us a little bit about Applitools and kind of what you’re expecting out of DevOps World during the next couple of days.

James: Yeah, sure. So Applitools, for those who aren’t familiar, we essentially exist to make sure the UI is visually perfect on every device, operating system, screen, mobile, desktop and so that’s what we do. And so, we actually are pretty unique entity here at Jenkins World this year. And, you know, what do we want out of it? I mean, the theme this year is transformed, as you know. And essentially, the issue or the problem that we solve for pretty much everyone in this room is digital transformation. So, we tie in really nicely in, you know, what digital transformation, I know it’s an industry buzzword, but really what that means, in a practical sense, is, you know, the UI for most brands in the world typically exists now in 500 to 1000 variations at this very moment in time. I mean, you think about your…or just our everyday lives constantly coming across broken UI and there is no brand spared. The largest brands in the world routinely have broken UI and it’s…you know, that’s really what we’re here to transform as we can get beyond this now. We built technology at Applitools that essentially solves for that problem and ensures a visually perfect world, as the t-shirt says.

Jayne: But you know the UI is really the user experience, right?

James: Yeah.

Jayne: So, you know, the influence of the interface and how the user perceives that often shapes the perception of the product, right? So, what’s your experience? What are users really looking for in terms of their UI experience?

James: Oh, I mean, I think, fundamentally, they want it to work. I mean, honestly, like you get…we’re to a point where this transformation and the number of iterations of the UI is so overwhelming, R&D and Dev teams simply can’t get up. I know in some sense of they’ve given up and honestly, customers see that. Now, the problem with that approach is that a customer can switch on a dime. I mean, I won’t name brands right now, but I was just trying to book an airline ticket and I literally had a responsive design issue on my mobile device. I could not check out. The button…that it passed the functional tests, right, from a test automation discussion. I was clicking the button to try to buy the ticket and it literally was not working, and I needed to get it done. I just switched to another airline. So, implications are pretty clear. You can have a UI issue or a bug that just looks bad for the brand, like doesn’t cast a great impression but more importantly, you can literally break the customer experience to the point where they just switch on the dime.

So, you know, we’re in a world and a situation where we can’t just give up any more and there’s no need to give up. Like you can actually solve for this problem through visual AI, which is what we do. We’ve actually created a technology that in real time, at scale, inspects the UI, spots the difference between what should be and what is, alerts the R&D team either pre-production [inaudible 00:03:42] monitoring and allows them to fix it. And so, it’s been a huge step forward for the industry and really completes that test automation chain, if you will, right? Because the UI…the reason the UI is such a problem is it exists outside a lot of the automation frameworks we’ve built up, right? And so, we no longer have to suffer this way.

Jayne: And I think that’s awesome. And I think the example that you gave really shows that it also can affect the bottom line.

James: Oh, absolutely.

Jayne: And so, it’s not just pretty, right? It’s not superfluous. It actually does have a strong influence on…

James: Huge influence.

Jayne: …brand recognition and everything that’s associated with that, which is fantastic. So, we’re here at DevOps World, so do you guys have any announcements that you want to, kind of, sneak peek us with?

James: Yeah. I know, so I think, probably the most exciting thing, and we just announced it a couple days ago, is the world’s first UI version control system. And so, let me explain what that means. Essentially, we have integrated with GitHub and we are…for the first time ever, you can actually tie your merge, right, your branch, your fork, your merge into your GitHub Repo to the UI directly, so you’re seamlessly linking the UI to that particular code merge, and tracking that history, you know, forever. And so now, you’re in a situation where if there’s a UI issue, you can tie it back to what merge caused that issue.

Jayne: Oh, that’s awesome.

James: And you have a history of, “Okay, we made these changes, who made these changes? Why do we make these changes? What exactly did the UI look like before these changes and what branch or fork was, you know, the one we need to fall back to?” And so, that’s just, I mean, it’s actually a really extraordinary…in simple concept. But for us to have the visual tied to that code build is transformative for how we deal with this problem I was just describing around digital transformation. So, that’s a big announcement and it’s really, really exciting.

Jayne: That’s extremely exciting and again, sometimes we don’t think of the UI as part of the Repo and being able to tie that together and trackback makes a lot of sense.

James: Exactly.

Jayne: So, thank you for spending some time.

James: You’re welcome.

Jayne: Really looking forward to the rest of the conference.

James: Yeah, absolutely.

Jayne: We’ve just kicked off a little while ago and really wish you a lot of luck with your great announcement. Thanks.

James: Thank you very much. Thank you, DevOps.com.

The post How to improve Customer Experience with Visual AI and Applitools appeared first on Automated Visual Testing | Applitools.

Next week, we’re joining IT and DevOps leaders, practitioners and friends at the Marriott Marquis in San Francisco for DevOps World | Jenkins World.

What is DevOps World | Jenkins World? DevOps World | Jenkins World is an international event that provides opportunities to learn, explore, network and help shape the future of DevOps and Jenkins continuous integration server. This year, it will be hosted at two separate locations: San Francisco, California from September 16-19, and Nice, France from October 22-25.

For the next few days, we’ll explore, share and learn about software automation, DevOps, performance metrics, security and more. Keynote speakers include Sacha Labourey, Christina Noren and Kohsuke Kawaguchi from CloudBees, Dr. Nicole Forsgren from DORA (that’s DevOps Research and Assessment, not a kid’s TV show), and Justin Graham from Amazon Web Services. We’re certain the event will offer expansive themes and takeaways, and look forward to gleaning new insights and advice from these DevOps leaders.

Don’t miss Angie Jones’s talk: “The Build That Cried Broken:  Building Trust in Your Continuous Integrations Test“. Angie’s talk will take place on Wednesday, Sept 19 @ 2:30 pm, at the Golden Gate C1-3 venue.

In her talk, Angie will cover the following topics:

• How to build stability within your CI tests
• Tips for managing CI tests that are failing with just cause
• How to influence the perception of the credibility of the tests among team members

Be sure to check it out!

We are also scheduled to present on Monday, September 17. James Lamberti, CMO, will discuss “The Missing Link in the Dev-Test-DevOps Chain: Intro to AI-powered Visual Testing and Monitoring.” Join us at 6 p.m. as he shares how to address the biggest challenges with testing and monitoring.

As Gold Sponsors, we’re happy to support DevOps World. You can find us at booth #111 during exhibit hours. We’re excited to share the latest visual testing and monitoring techniques that support automation and improved visual UI testing through the Application Visual Management (AVM) approach.

We’re also looking forward to sharing the latest from Applitools Eyes, our Automated Visual AI Testing Platform. Some of the latest features include UI Version Control that lets you build software smarter by capturing the entire visual history of your apps, so you have a record of what’s been changed, by whom, and when — and what should be rolled back if necessary.

Applitools Eyes offers UI baseline management, team collaboration, expanded third-party tool integrations and extensive analytics dashboards – eliminating the stress of manual visual testing through it’s AI features. This helps ensure a visually perfect user experience on every single browser and device. Stop by booth #111 to learn more.

We hope to see you there!

If you don’t get a chance to connect with us a Jenkins World, reach out or sign up for a free Applitools account.

The post DevOps World | Jenkins World San Francisco: Meet Us There, Speaking and Sponsoring! appeared first on Automated Visual Testing | Applitools.

Improving speed and ease of use, the new Applitools Eyes open-source plugin allows to perform visual test analysis and maintenance directly from the Jenkins Build Report. 

Listen to Adam Carmi – Applitools co-founder & CTO – present how to implement and use the new open-source Jenkins plugin:

JENKINS WORLD – Santa Clara, CA – September 14, 2016 – Applitools announced today the release of its new open-source Jenkins plugin for automated visual testing at Jenkins World, the annual gathering of DevOps practitioners using open source Jenkins. The conference held at the Santa Clara Convention Center is the largest gathering of Jenkins users in the world and is a multi-day event comprised of sessions, workshops, training and other learning opportunities.

The new Jenkins plugin adds the visual tests results from Applitools Eyes directly in the Jenkins Build Report. Integrating Applitools Eyes results into the Jenkins Build report allows users to see a high-level view of the detected visual differences as part of the Jenkins job, drill down and inspect specific differences up-close, and maintain test baselines directly from the Jenkins Build Report.

“We’ve had many requests from customers to enhance the integration of Applitools Eyes with Jenkins and similar CI systems. This new open-source plugin allows our customers to perform all the test analysis and maintenance activities directly from Jenkins, helping them further speed up release processes, in addition to automating visual testing”, said Adam Carmi, Applitools Co-founder & CTO.

“Jenkins World is the perfect place for Applitools to announce their new Jenkins plugin for automated visual testing, which pushes the envelope of automation and moves us one step closer to fully-automated CI processes and faster release cycles,” said Kohsuke Kawaguchi, CTO of CloudBees and founder of the Jenkins project.

Additional Resources:

• Link to the Applitools Eyes Jenkins plugin On Jenkins Wiki
• Learn more about Automated Visual Testing with Applitools
• Open your free Applitools Eyes account
• How-to video, presenting the new plugin:

To read more about Applitools’ visual UI testing and Application Visual Management (AVM) solutions, check out the resources section on the Applitools website. To get started with Applitools, request a demo or sign up for a free Applitools account.

The post Applitools Releases Open-Source Jenkins Plugin for Automated Visual Testing appeared first on Automated Visual Testing | Applitools.

Must-see webinar for test automation professionals, presented by Automation expert Dave Haeffner – now available online, on-demand!

Learn how to build simple and powerful automated cross-browser tests, covering visual testing and functional regressions, and configured to run automatically through the use of a Continuous Integration (CI) server.

Watch this step-by-step webinar, presented by Test Automation expert Dave Haeffner (author of Elemental Selenium and The Selenium Guidebook), and learn how to: 

• Get started off on the right foot with Selenium by performing hundreds of assertions with just a few lines of code
• Run your automated web tests on any browser and operating system combination
• Automate your test runs with a Continuous Integration server

Watch it now:

Dave’s slides can be found here: 

To read more about Applitools’ visual UI testing and Application Visual Management (AVM) solutions, check out the resources section on the Applitools website. To get started with Applitools, request a demo or sign up for a free Applitools account.

The post Awesome Test Automation Made Simple – presented by Dave Haeffner appeared first on Automated Visual Testing | Applitools.

Automating Your Test Runs with Continuous Integration 

 

You’ll probably get a lot of mileage out of your automated tests if you run things from your computer, look at the results, and tell people when there are issues in the application. But that only helps you solve part of the problem.

The real goal in test automation is to find issues reliably, quickly, and automatically – and ideally, in sync with the development workflow you’re a part of.

To do that we need to use a Continuous Integration server.

A Continuous Integration Server Primer

A Continuous Integration server (a.k.a. CI) is responsible for merging code that is actively being developed into a central place (e.g., “trunk” or “master”) frequently (e.g., several times a day, or on every code commit, etc.) to find issues early so they can be addressed quickly — all for the sake of releasing working software in a timely fashion.

With CI, we can automate our test runs so they can happen as part of the development workflow. The lion’s share of tests that are typically run on a CI Server are unit (and potentially integration) tests. But we can very easily add in our recently written Selenium tests.

There are numerous CI Servers available for use today, most notably:

• Bamboo
• Jenkins
• Solano Labs
• TravisCI
• etc.

Let’s step through an example of using Jenkins on CloudBees.

An Example

Jenkins is a fully functional, widely adopted, and open-source CI adn CD (Contibuous Delivery) server. Its a great candidate for us to step through. And DEV@cloud is an enterprise-grade hosted Jenkins service offered by CloudBees, the enterprise Jenkins company. It takes the infrastructure overhead out of the equation for us.

1. Quick Setup

We’ll first need to create a free trial account, which we can do here.

Once logged in we can click on Get Started with Builds from the account page. This will take us to our Jenkins server. We can also get to the server by visiting http://your-username.ci.cloudbees.com. Give it a minute to provision, when it’s done, you’ll be presented with a welcome screen.

NOTE: Before moving on, click the ENABLE AUTO-REFRESH link at the top right-hand side of the page. Otherwise you’ll need to manually refresh the page to see results (e.g., when running a job and waiting for results to appear).

2. Create A Job

Now that Jenkins is loaded, let’s create a Job and configure it to run our tests.

1. Click New Item from the top-left of the Dashboard
2. Give it a descriptive name (e.g., Login Tests IE8)
3. Select Freestyle project
4. Click OK

This will load a configuration screen for the Jenkins job.

 

3. Pull In Your Test Code

Ideally your test will live in a version control system (like Git). There are many benefits to doing this, but the immediate one is that you can configure your job (under Source Code Management) to pull in the test code from the version control repository and run it.

1. Scroll down to the Source Code Management section
2. Select the Git option
3. Input the Repository URL (e.g., https://github.com/tourdedave/getting-started-blog-series.git)

Now we’re ready to tell the Jenkins Job how to run our tests.

4. Add Build Execution Commands

1. Scroll down to the Build section
2. Click on Add Build Step and select Execute Shell

In the Command input box, add the following commands:

export SAUCE_USERNAME="your-sauce-username"
export SAUCE_ACCESS_KEY="your-sauce-access-key"
export APPLITOOLS_API_KEY="your-applitools-api-key"
gem install bundler
bundle install
bundle exec rspec

Since our tests have never run on this server we need to include the installation and running of the bundler gem (gem install bundler and bundle install) to download and install the libraries (a.k.a. gems) used in our test suite. And we also need to specify our credentials for Sauce Labs and Applitools Eyes (unless you decided to hard-code these values in your test already – if so, then you don’t need to specify them here).

5. Run Tests & View The Results

Now we’re ready to save, run our tests, and view the job result.

• Click Save
• Click Build Now from the left-hand side of the screen

When the build completes, the result will be listed on the job’s home screen under Build History.

You can drill into the job to see what was happening behind the scenes. To do that click on the build from Build History and select Console Output (from the left navigation). This output will be your best bet in tracking down an unexpected result.

In this case, we can see that there was a failure. If we follow the URLs provided, we can see a video replay of the test in Sauce Labs (link) and a diff image in Applitools Eyes.

The culprit for the failure here wasn’t a failure of functionality, but a visual defect with the image on the Login button.

A Small Bit of Cleanup

Before we can call our setup complete, we’ll want a better failure report for our test job. That way when there’s a failure we won’t have to sift through the console output for info. Instead we should get it all in a formatted report. For that, we’ll turn to JUnit XML (a standard format that CI servers support).

This functionality doesn’t come built into RSpec, but it’s simple enough to add through the use of another gem. There are plenty to choose from with RSpec, but we’ll go with rspec_junit_formatter.

After we install the gem we need to specify some extra command-line arguments when running our tests. A formatter type (e.g., --format RspecJunitFormatter) and an output file for the XML (e.g., --out results.xml). And since this type of output is really only useful when running on our CI server, we’ll want an easy way to turn it on and off.

# filename: .rspec

<% if ENV['ci'] == 'on' %>
--format RspecJunitFormatter
--out tmp/result.xml
<% end %>

Within RSpec comes the ability to specify command line arguments that are used frequently in a file (e.g., .rspec) that lives in the root of the test directory. In it we specify the new commands we want to use and wrap them in a conditional that checks an environment variable that denotes whether or not the tests are being run on a CI server (e.g., if ENV['ci'] == 'on').

Now it’s a small matter of updating our Jenkins job to consume this new JUnit XML output file by adding a post-build action to publish it as a report.

Then we need to tell the Jenkins job where the XML file is. Since it ends up in the root of the test directory, we can just specify the file extension with a wildcard.

Lastly, we need to update the shell commands for the build to set the ci environment variable to on.

Now when we run our test, we’ll get a test report which states which test failed. And when we drill into it, we get the URLs for the jobs in Sauce Labs and Applitools Eyes.

One More Thing: Notifications

In order to maximize your CI effectiveness, you’ll want to send out notifications to alert your team members when there’s a failure.

There are numerous ways to go about this (e.g., e-mail, chat, text, co-located visual cues, etc). And thankfully there are numerous, freely available plugins that can help facilitate whichever method you want. You can find out more about Jenkins’ plugins here.

For instance, if you wanted to use chat notifications and you use a service like HipChat or Slack, you would do a plugin search and find one of the following plugins:

After installing the plugin for your chat service, you will need to provide the necessary information to configure it (e.g., an authorization token, the channel/chat room where you want notifications to go, what kinds of notifications you want sent, etc.) and then add it as a Post-build Action to your job (or jobs).

Now when your CI job runs and fails, a notification will be sent to the chat room you configured.

Outro

If you’ve been following along through this whole series, then you should now have a test that leverages Selenium fundamentals, that performs visual checks (thanks to Applitools Eyes), which is running on whatever browser/operating system combinations you care about (thanks to Sauce Labs), and running on a CI server with notifications being sent to you and your team (thanks to CloudBees).

This is a powerful combination that will help you find unexpected bugs (thanks to the automated visual checks) and act as a means of collaboration for you and your team.

And by using a CI Server you’re able to put your tests to work by using computers for what they’re good at – automation. This frees you up to focus on more important things. But keep in mind that there are numerous ways to configure your CI server. Be sure to tune it to what works best for you and your team. It’s well worth the effort.

To read more about Applitools’ visual UI testing and Application Visual Management (AVM) solutions, check out the resources section on the Applitools website. To get started with Applitools, request a demo or sign up for a free Applitools account.

The post Automating Your Test Runs with Continuous Integration — CI Series by Dave Haeffner: Part 3/3 appeared first on Automated Visual Testing | Applitools.