We’re honored to be co-authors with Sogeti on their “State of AI applied to Quality Engineering 2021-22” report. In the latest chapter, learn how you can use Visual AI today to release software faster and with fewer bugs.

In the world of software development, there is a very clear trend – greater application complexity and a faster release cadence. This presents a massive (and growing) challenge for Quality Engineering teams, who must keep up with the advancing pace of development. We think about this a lot at Applitools, and we were glad to be able to collaborate with Sogeti on the latest chapter of their landmark “State of AI applied to Quality Engineering 2021-22” report, entitled Shorten release cycles with Visual AI. This chapter is focused around this QE challenge and offers a vision for how Visual AI can help organizations that have not yet adopted it – not far in the future but today.

What is Visual AI

Visual AI is the ability for machine learning and deep learning algorithms to truly mimic a human’s cognitive understanding of what is seen. This may seem fantastical, but it’s far from science fiction. Our own Visual AI has already been trained on over a billion images, providing 99.9999% accuracy, and leading digital brands are already using it today to accelerate their delivery of innovation.

Leverage Visual AI to Shift Left and Deliver Innovation Faster

Visual AI can be used in a number of ways, and it may be tempting to think of it as a tool that can help you conduct your automated end-to-end tests at the end of development cycles more quickly. Yes, it can do that, but its biggest strength lies elsewhere. Visual AI allows you to shift left and begin to conduct testing “in-sprint” as part of an Agile development cycle.

Testing “in-sprint” means conducting visual validation alongside data validation and gaining complete test coverage of UI changes and visual regressions at every check-in. Bottlenecks are removed and releases are both faster and contain fewer errors, delivering an uncompromised user experience without jeopardizing your brand.

Teams that incorporate automated visual testing throughout their development process simply release faster and higher quality software.

How Can You Use Visual AI Today?

Wondering how you can move your organization or your team over to the left side of the bar charts above? Fortunately, it’s not hard to get started, and this chapter from Sogeti is an excellent place to begin. Keep reading to learn more about:

• When you should visually test your UI (and why it’s important)
• How to automate UI validation with Visual AI (including the three biggest challenges and how to overcome them)
• Different Visual AI comparison algorithms
• How Visual AI significantly reduces test creation and maintenance time while increasing coverage
• Streamlining analysis of test results and root cause analysis
• Using Visual AI for end-to-end validation
• Validation at check-in with Visual AI
• How Visual AI is revolutionizing cross browser testing

Deliver Quality Code Faster with Visual AI

Ultimately, what we’re all looking for is to be able to deliver quality code faster, even as complexity grows. Keeping up with the growing pace of change can feel daunting when you’re relying on traditional test automation that only scales linearly with the resources allocated – AI-powered automation is the only way to scale your team’s productivity at the pace today’s software development demands.

Applitools’ Visual AI integrates into your existing test automation practise and is already being used by the world’s leading top companies to greatly accelerate their ability to deliver innovation to their clients, customers and partners, while protecting their brand and ensuring digital initiatives have the right business outcomes. And it’s only getting better. Visual AI continues to progress as it advances the industry towards a future of truly Autonomous Testing, when the collaboration between humans and AI will change. Today, we’re focused on an AI that can handle repetitive/mundane tasks to free up humans for more creative/complex tasks, but we see a future where Visual AI will be able to handle all testing activities, and the role of humans will shift to training the AI and then reviewing the results.

Check out the full chapter, “Shorten release cycles with Visual AI,” below.

The post How Visual AI Accelerates Release Velocity appeared first on Automated Visual Testing | Applitools.

Applitools was invited to share our expertise in applying AI to quality engineering, and we’re honored to be co-authors of this comprehensive report by Sogeti.

Sogeti has just released the first section of their State of AI applied to Quality Engineering 2021-22 report, including two chapters co-authored by Applitools. The report is a detailed examination of the current state of artificial intelligence in the field of quality engineering. It centers around a key question – how can AI make our quality validation smarter? In the words of the executive introduction:

This report aims to assist you in understanding the potential of AI and how it can help improve the quality, velocity, and efficiency of your quality engineering activities.

As one of the pioneers in the application of AI to quality engineering through Visual AI, we were honored to be asked to participate in this report and share our expertise. We co-authored several chapters, including two that have been released today in the first section.

Chapter 1: Quality Engineering and recent trends

In this chapter, you’ll get an overview of the business and technical environment which has led us to where we are today and the current need for assistance from AI. It discusses the shortcomings of traditional testing practices and the emergence of modern quality engineering. What does a successful Quality Engineer do today? What are the challenges faced? What is the future of quality engineering, and what role could AI play in that? Check out this opening chapter for a great introduction into the topic of AI in QE. 

Chapter 2: Getting started with AI

This chapter digs a little deeper into how you can get started in your journey with AI. Moshe starts by relating a personal story about a customer service experience that left him frustrated. How can we use AI to eliminate waste from our days and spend more time on quality engineering and address issues before they impact end users? The chapter goes on to cover the difference between routine and error-prone tasks and opens up the discussion of how we can optimize each type. You’ll also get some great info on how to define AI, understand possible use cases, and thoroughly research your options. Head over to the second chapter to read more.

In chapter 3 and chapter 4, you can explore further with technical deep dives into machine learning and deep learning. 

Check out the Full Report

Sogeti has put together a strong report on this important topic and we’re excited to share the opening section with you today. Starting in September, you can expect to find new sections released bi-weekly, including another chapter from Applitools that will be out in the coming months. To learn more, check out the full “State of AI applied to Quality Engineering 2021-22” report. 

The post The “State of AI applied to Quality Engineering 2021-2022” Report Released appeared first on Automated Visual Testing | Applitools.

Google Firebase Test Lab is a cloud-based app-testing infrastructure. With one operation, you can test your Android or iOS app across a wide variety of devices and device configurations, and see the results—including logs, videos, and screenshots—in the Firebase console.

Firebase Test Lab runs Espresso and UI Automator 2.0  tests on Android apps, and XCTest   tests on iOS apps. Write tests using one of those frameworks, then run them through the Firebase console or the gcloud command line interface.

Firebase Test Lab lets you run the following types of tests:

• Instrumentation test: A test you write that allows you to drive the UI of your app with the actions you specify. An instrumentation test can also make explicit assertions about the state of your app to verify correct functionality using AndroidJUnitRunner APIs. Test Lab supports Espresso and UI Automator 2.0 instrumentation test frameworks.
• Robo test: A test that analyses your app’s interface and then explores it automatically by simulating user activities.
• Game loop test: A test that uses a “demo mode” to simulate player actions in game apps.

How do I validate the Visual UI in Multiple Devices?

As with all web and mobile applications. Applitools offers an easy, consistent way to collect visual data from multiple device types running different viewport sizes. In the rest of this article, you will run through a demonstration for using Applitools with Google Firebase Test Lab.

Demo

In this Demo I have choose a simple “Hello World” app,  and to keep you running we already have an example Espresso Instrumentation Test you can find the Complete Project here https://github.com/applitools/eyes-android-hello-world 

Prerequisites

I know you have looked into the GitHub repo. Let’s just get few more prerequisites installed and make sure they are ready to use and deep dive. Make sure you have installed and/or configured the following:

• Java Installed
• JAVA_HOME environment variable is set to the Java SDK path
• Android Studio IDE

Installing the Android Studio

Now let’s install Android Studio / SDK so that you can run the test script on an emulator or real device. You could install the Android SDK only but then you have to do additional advanced steps to properly configure the Android environment on your computer. I highly recommend installing the Android Studio as it makes your life easier.

Download the   Android Studio executable. Follow the steps below to install locally on your computer:

Run the Visual UI Test locally

1. Get the code:

• Option 1: git clone https://github.com/applitools/eyes-android-hello-world
• Option 2: Download it as a Zip file and unzip it.

2. Import the project into Android Studio

The Script

Let’s look at the Instrumented test ExampleInstrumentedTest under androidTest.    

https://github.com/applitools/eyes-android-hello-world/blob/master/app/src/androidTest/java/com/applitools/helloworld/android/ExampleInstrumentedTest.java

Before we run the test on Firebase lets run it on local emulator.

1. Configure a local device Emulator from the AVD Manager in your android studio
2. Insert Your Applitools API KEY in the test
3. Launch the SimpleTest () from the ExampleInstrumentedTest.java. This will launch the Instrumentation test in the emulator.

That’s pretty easy isn’t it? Applitools will now capture each screen where the eyes.CheckWindow() is called and create a baseline on the first run.

Once the test completes, you can analyze the test results on the Applitools dashboard.

Firebase

Now let’s run the test on Firebase devices. To do this we need first need an account so let’s get that

Step1: Navigate to https://firebase.google.com/ and click on sign in

Step2: Click on Go to Console, Navigates to the Console Dashboard

Step3:  Create a Project, once you create a project, all good to explore the dashboard and see through all the features available here.

 Step 4: Lets Add the Configurations in android studio to run the tests

• Click on Run and select the edit configurations.
• Click on the + button to create a new android launch/debug configuration based on templates.
• Select the Android Instrumentation Tests > In the Module select app >
  Test as Class > com.applitools.helloworld.android.ExampleInstrumentedTest
• Now in the Deployment Target Options select Firebase Test lab Device Matrix

Step 5: Sign in with Google Firebase account and click ok

Step 6: Re-open the edit configurations

Now you can see the configure settings for Matrix configuration and cloud project

Select your project and add one or more custom devices from list of 150. For now let’s add 2 devices, Platform Android 9.x, API Level 28 (pie), Locale, Orientation.

We will use these devices to run our Instrumentation test on Firebase.

That’s it we are all good to run our test on the Firebase

Click on Run Example Instrumented Test this will now execute you tests on the devices you have selected on Firebase.

 Let’s Go back to The Test lab on Firebase and you can see your tests running over there Parallelly with visual comparison checks done on Applitools AI Platform.

Conclusion

Applitools allows you to test your mobile app by running it on any device lab. Google Firebase allows a streamline platform for developers (build) and quality engineers (test) to run tests on any device configuration. The integration make it easier to use the best of platforms for best quality applications.

For More Information

• CI Test Automation Strategy
• Test Automation for Android Wearable Devices with Appium
• Test Automation as a Key Enabler for High-performing Teams – with Angie Jones and Google’s Nicole Forsgren
• Master Appium in 4 Steps and 11 Tutorials
• Sign up for a free Applitools account
• Request an Applitools demo

The post How Do I Test Mobile Apps At Scale With Google Firebase TestLab And Applitools? appeared first on Automated Visual Testing | Applitools.

How Do You Build A Culture of Quality Excellence?

Greg Sypolt, VP of Quality Assurance at EverFi, led an hour-long webinar for Applitools titled “Building A Culture of Quality Excellence – Understanding the Quality Maturity Rubric.” Let’s review his key points and his core slides.

About Greg Sypolt

Greg has been writing blogs about his experiences as a quality advocate inside his company. He wants to learn how to make testing more efficient and share what he learns with others. He loves to geek out about testing and technology, but he’s also a sports fan, he loves doing DIY projects, and he’s a husband and father of three.

Overview

Greg’s webinar focuses on you – someone who is looking to improve the quality culture at your organization. No matter where you are on the journey, from just starting out to making significant progress, Greg has useful thoughts and ideas for you.  His webinar breaks into several key pieces:

• Vision
• Journey
• Three Pillars of App Quality Excellence
• How To Measure The Quality Maturity Rubric
• Building A Quality Platform
• Reaping The Savings

Vision

Greg’s key message starts with a key idea: Quality is a journey, not a destination. Your tools and software constantly evolve, and your processes and procedures evolve as well.

How do you build your culture of quality excellence? To start, Greg says, envision an organization where each member has the mindset that he, she, or they own quality. This vision differs drastically from the legacy idea of a “QA team”. By getting on that path towards broad quality ownership, Greg says, each team member can focus on becoming the voice of quality for the organization.

At the same time, Greg says, understand your long-term goals. You want to build an organization where quality aligns with your organization’s other goals. You want to provide both speed and efficiency, in addition to reliability. Your organization likely needs a higher level of collaboration. And, by speed, he means focusing on the speed of testing – making sure that your actual tests run quickly. Quality should not be your bottleneck.

Quality Journey Objectives

Next, Greg shows the following slide:

Greg describes each of these steps in turn.

• Everyone understands how we make and test things. In this step, you have a good understanding that design requirements involve validation requirements. You know that you want unit tests, API tests, integration tests, and UI tests. You want to handle functional cases and failure cases. These cases need to be part of the design.
• Building a quality mindset. Here, you recognize the importance of quality in every aspect of your process to produce a quality product.
• Understanding our current testing coverage. Provide transparency to your team about what you cover and what you do not. Prioritize areas for test development.
• Reduction of manual testing, advocating for automating the right things. Slow, inconsistent and inaccurate manual tests impede your testing speed. But prioritize. Know which tests benefit from automation.
• Build a balanced testing portfolio. Understand how well you cover behavior across unit, API, integration, UI, performance, and security tests. Know your deficiencies and address them.
• Shift-left mindset by embracing quality in every stage of the CICD pipeline. As you build, test. Ensure that your pipeline gates work well to check each code build against expected behavior.
• Transitioning to exploratory charters and fully automated testing. This step gets you to automating the right things and beginning to expand your thinking to uncovered cases.
• Building in quality velocity for developers. Now you deliver ways to help your development team create with a quality mindset by designing with test, testability, and validation results in mind.
• Quality visibility and accountability. Create the tools and the feedback loop to show discovered defects and their origin – not as a path to punishment but to aid improvement.
• Remove the traditional QA silo mentality. As part of all the other steps, quality becomes a team metric, rather than the responsibility of a subgroup tasked with assurance.
• Not a destination. The journey continues – improvements continue along your path.

Greg’s Journey As An Example

Greg shows some data from his experience at his previous company. Greg had worked there for five years.  When he started, the company had no test automation. Each release took weeks to accomplish.

At the end of his journey, the company:

• Ran 4000 or more builds per week
• Regularly generated 600 Selenium, 55 Visual, and 75 Integration automated tests
• Ran an average of 5,000,000 or more tests per month with an average 99.2% pass rate
• Authored tests in less than 15 minutes and ran all their tests in less than 5 minutes.

By creating all this test infrastructure to go along with the software development builds, Greg’s team created the infrastructure to validate application behavior through test automation. They also built the tools to automate test generation. Finally, they created a discipline to build tests that would run quickly with comprehensive coverage, so testing would allow those 4000+ weekly builds.

Clearly, the process before and after required an evolution at the company. He is working with his team at Everfi with the goal of achieving a comparable outcome.

The Three Pillars of Application Quality Excellence

Next, Greg speaks about the core pillars of building a culture of quality.

First, Greg describes “Quality over quantity.” The first step here is to test the right things, but not everything. Every line of test code you write becomes test code you need to maintain. New features get added, and your test code will likely have to change. You need your developers to learn to think from an end-user’s perspective – how a user will use the application? Analytic tools can help your team understand the paths that users take through  your application to know what’s important. That data can help you get a robust test infrastructure that targets how your users use your products.

Quality over quantity also focuses your team on transparency for quality. If a developer makes a change and kicks off a build, the build result provides immediate feedback to the developer and the team if the added feature causes test failures. Hold everyone accountable for the quality of their code.

Second, Greg describes “Work smarter not harder.” At Greg’s previous company, they had a dedicated developer solutions team providing self-service solutions for testing. That team made it easy for developers to build the app and tests at the same time. They also built a central repository for metrics, so testing results across behavior, visual, and integration tests could be viewed and analyzed in one place. They worked on tools to author tests more quickly and more effectively. The result made it easier to create the right tests.

They also looked at ways to include more than just developers in helping to both understand the tests and to help create tests.

Third, Greg explains what he means by “Set the right expectations.” He makes it clear that you need to set the right expectations with the development team. When developers begin to develop tests, they need to understand what you expect from them – both from a coverage as well as velocity perspective. You need to make sure that everyone understands who can ensure quality and performance of the product – and it’s the people who write the code who have the greatest influence on the quality of that code.

Greg also makes it clear that the you need a clear plan to move to a more mature quality structure. Everyone needs to know what is expected, what will be measured, and what outcomes the team wants to achieve.

Measuring Quality Maturity Rubric

At this point, Greg shares his rubric for analyzing quality maturity.

He mentions that the metrics involve the people on your teams, the technology you use, and the processes you put in place. Your people need to be ready and organized appropriately. You require the right technology to increase your automation. And, you need to put processes in place that allow you to succeed at improving your quality maturity level.

As a preface, Greg describes these metrics and this rubric as based on his experience. He expresses an openness to update these based on the experience of others.

The Quality Maturity Rubric

His full rubric involves 23 different maturity attributes that each have four range values:

• Beginner – getting started on maturity
• Intermediate – taking steps to organize people, use technology, or apply processes
• Advanced – organizing effectively to achieve the maturity measure outcome
• Expert – having automated the process to be achieved with people used at their most efficient.

Greg goes through several attributes in detail, giving examples of beginner through expert for each.  For example:

This table shows the “Culture” attribute.

• Beginner – No shared quality ownership, siloed development and QA, losing sight of the larger quality picture.
• Intermediate – Identify and define specific levels of quality involvement for individuals on teams, enable shared responsibilities
• Advanced – Teams are finding problems together
• Expert – Machines are identifying problems

To use this table, compare your organization to the description of each value that most closely matches yours. Give yourself a “1” if you’re a beginner, “2” if you’re Intermediate, “3” if you’re advanced, and “4” if you achieved expert.

Greg has created tables for each of the attributes. For example:

For the Environment attribute, the Advanced level involves automated builds and staging, with push-button deployment.

Using the rubric, you go through each attribute and assign a numeric value based on how close you are to one of the levels.

As you go through the analysis, you can also use the table to set your goals for improvement over time, as you look to increase the quality maturity of your organization.

Greg posted his grading rubric online – you can find it at:

http://bit.ly/qmc-grading-rubric

Once you use the grading rubric, you can start to figure out the next two steps:

• Create an improvement map. You can’t move everything all at once. Focus on the key attributes that matter. Greg points out that culture allows you to move everything else. Figure out where you can go with culture maturity.
• Move to implementation. Once you know what you want to do, move forward with appropriate steps. Do you have the right people? Do you have process changes you need to deploy? New technology? Move forward in steps appropriate to your organization.

Quality Roles

Now that you have people, processes and technology thought through, along with your approach to maturity, it’s time to think about your people and their skills.

Greg presents a map of QA role clarity that helps you think about your existing technical skills among your team. Greg created this table based on his experiences in quality engineering.

1. Level 1 is a QA specialist, who focuses on manual and exploratory test
2. The second level is an automation engineer, who has the Level 1 skills plus the ability to develop automation test scripts.
3. At Level 3 you find an Industry SDET (software development engineer in test) who does the work of the first two levels, plus can write code and develop test algorithms.
4. Level 4 defines the need of EverFi – an EverFi SDET. In addition to the first three levels, an EverFi SDET must be proficient with DevOps. Greg hopes to get his team to this level in the next 12 to 18 months. A DevOps SDET can help make the development team proficient integrating test with development.
5. The top level, Level 5 – the Next Gen SDET, incorporates the prior four levels. On top of that, a Next Gen SDET brings proficiency in security, data science, and machine learning. This level is more aspirational. Greg expects that, over time, more quality engineers will obtain these skills.

Greg sees this table as another rubric you can use to evaluate your people. You can evaluate where you are today. And, you can start to think about skills you want to add to your team. The people on your team will help you execute your vision, so you need to know where you are and where you want to go.

You can look to hire people with these skills. You likely can find engineers with Level 3 skills of SDET. More than likely, though, you, like Greg, will be building these skills among your team over time.

Building A Quality Platform

Once you have thought through your quality maturity, reviewed your existing processes and technologies and begun to evaluate your team, Greg wants you to think about your current and future states as a “quality platform”.

He reminds you that your quality platform serves key outcomes:

• Building a culture that embraces quality at every stage from intake, discovery, execution and release
• Enabling and driving continuous improvement and adoption of quality practices
• Giving teams the ability to lead with a sense of purpose, openness and trust.

The combination of people, processes, and technologies that make up your quality platform can help you deliver major quality improvements.

At the core of your quality platform is the Developer Solutions team – working to create quality solutions among the software development team.

Next, you need data insights that help create visibility across the organization.

Third you develop turnkey solutions that simplify the deployment of key functions or processes. For example you can easily deploy Selenium or Cypress test automation through a set of well-defined code structures, and. You can easily build new tests and structures in your code. For example, at EverFi, Greg has deployed Applitools to easily add visual validation to tests – simplifying overall test development.

Fourth – your platform team serve as quality ambassadors. They represent quality practices and endorse effective change within the organization.

Fifth – you focus on functional values of the platform, making testing better and easier for everyone.

Lastly, you focus on non-functional behaviors, like monitoring the critical paths inside your application. You make sure to understand the critical paths and ensure they work correctly.

Reaping The Savings

Finally, Greg gets to the bottom line.

How does the quality platform deliver savings to your organization? Greg shows this handy table.

Each element of the quality platform contributes savings.

Greg gives the example of Data Insights providing cost savings because they help you know where you can add tests as well as providing data that can be consumed by the team. Developer Solutions helps you reduce the time to deploy pre-existing or new solutions. Functional tests can help improve your quality by ensuring you run tests on every pull request and get instant feedback, instead of waiting for savings. Or, a turnkey session with an external vendor can help you get up to speed quickly with the vendor’s technology.

Building Your Own Quality Platform

As Greg points out, moving to your own quality platform results in a journey of constant improvement. Your begin with your current state, envision a future state, and move to deliver that future state through people, technology, and process improvements. Each step along the way you end up with measurable savings.  

Here’s to your journey!

For More Information

• Measure Your Test Automation Maturity
• How To Ace High-Performance Test for CI/CD
• The Impact of Visual AI on Test Automation Report
• How Do You Catch More Bugs In Your End-To-End Tests?
• Sign up for a free Applitools Account
• Request an Applitools demo

The post How Do I Build A Culture of Quality Excellence? appeared first on Automated Visual Testing | Applitools.