EMA (Enterprise Management Associates) recently released a report titled “Disrupting the Economics of Software Testing Through AI.” In this report, author Torsten Volk, Managing Research Director at EMA, discusses the reasons why traditional approaches to software quality cannot scale to meet the needs of modern software delivery. He highlights 5 key categories of AI and 6 critical pain points of test automation that AI addresses. 

We sat down with Torsten to talk about the report and hear his insights on how AI is impacting Software Testing:

1. What’s wrong with the current state of testing? 
   
   Many software development teams are struggling to deliver on the promise of Agile/DevOps and are faced with numerous technical challenges, such as rising application complexity and explosion of browsers/devices. Multiply this by the constant drive for faster releases to deliver increased customer value – without the ability to hire additional quality control staff – and you can quickly see why the traditional approach to software testing can no longer scale to keep up. We need to optimize the process with AI to eliminate the mundane and repetitive tasks and ultimately ensure business success.

2. How does AI help with software testing?
   
   There are five key capabilities AI provides to help improve human efficiency: smart test creation, self healing, coverage detection, anomaly detection, and visual inspection. In the report, I discuss the six critical pain points where these capabilities are delivering ROI today: false positives, test maintenance, inefficient feedback loops, rising application complexity, device sprawl, and tool chain complexity.
   
   

   The 5 Key AI Capabilities

   Of the capabilities available, AI-driven visual inspection has the broadest reach and highest impact. This discipline aims to provide test engineers with an additional “pair of eyes,” leaving the engineer to focus on activities that really need human intelligence. It provides humans with the contextual information needed to accelerate their test and remediation efforts, recommending solutions wherever necessary and remembering human decisions.

3. You talk about visual inspection having the highest impact. What pain points does it address?
   
   Traditional pixel-based comparison attempts to perform visual inspection but is plagued with false positives. Training deep learning models to inspect an application through the eyes of the end-user removes a lot of the mundane repetitive tasks that cause humans to be inefficient. Due to its accuracy, AI-driven visual inspection can enable teams to create a stable suite of automated tests with reduced false positives. Visual inspection can be further leveraged to drive cross-browser/cross-device validation at scale.

4. How are people adopting AI? Do I need to hire AI experts or develop an AI practice? 
   
   The world’s top brands are already adopting AI to transform their software testing, but they’re focusing on the ROI gained from the speed and scalability of leveraging AI in the development process – not developing the AI itself. There are solutions on the market today that give you the ability to take advantage of fully trained models and several can be layered on top of your existing test automation. One of the solutions that seems widely adopted for visual inspection is Applitools, which is already trained on +1B images and delivers extremely high levels of accuracy.

5. How does the role of the developer / tester change with the introduction of AI?
   
   Developers and testers still need to make a decision about what and how something should be automated. AI minimizes the mundane and repetitive tasks, freeing the engineers to do more creative, interesting and valuable work.
   
   For example, what happens when a tester is reviewing thousands of webpages for accuracy? Some pages have differences, but they don’t matter. AI can filter those out and highlight only the subset that needs review.
   
   Auto-classification, grouping issues together, is another great example of being able to reduce your work and improve accuracy. If I’m looking at what looks like the same problem over and over again, I might miss a defect. Whereas if the AI categorizes and groups similar issues together it greatly reduces the possibility of a missed defect.

6. What’s the future state for software testing, and where do we start?
   
   Autonomous testing is the vision for the future, but we have to ask ourselves, why don’t we have an autonomous car yet? It’s because today, we’re still chaining together models, and models of models. We’re striving to get to the point where AI is taking care of all of the tactical and repetitive decisions and humans are thinking more strategically at the end of the process, where they are more valuable from a business-focused perspective. I would recommend starting with the mature solutions available today in the areas of visual inspection and self healing – keeping an eye on the future and ongoing evolution of AI. 

Thanks to Torsten for spending the time with us. If you would like to start implementing AI as part of your testing pipeline, sign up for a free Applitools account and start enjoying the benefits of Visual AI in minutes.

This article was originally published on SD Times.

The post Disrupting the Economics of Software Testing Through AI: An Interview with Torsten Volk appeared first on Automated Visual Testing | Applitools.

We’re honored to be co-authors with Sogeti on their “State of AI applied to Quality Engineering 2021-22” report. In the latest chapter, learn how you can use Visual AI today to release software faster and with fewer bugs.

In the world of software development, there is a very clear trend – greater application complexity and a faster release cadence. This presents a massive (and growing) challenge for Quality Engineering teams, who must keep up with the advancing pace of development. We think about this a lot at Applitools, and we were glad to be able to collaborate with Sogeti on the latest chapter of their landmark “State of AI applied to Quality Engineering 2021-22” report, entitled Shorten release cycles with Visual AI. This chapter is focused around this QE challenge and offers a vision for how Visual AI can help organizations that have not yet adopted it – not far in the future but today.

What is Visual AI

Visual AI is the ability for machine learning and deep learning algorithms to truly mimic a human’s cognitive understanding of what is seen. This may seem fantastical, but it’s far from science fiction. Our own Visual AI has already been trained on over a billion images, providing 99.9999% accuracy, and leading digital brands are already using it today to accelerate their delivery of innovation.

Leverage Visual AI to Shift Left and Deliver Innovation Faster

Visual AI can be used in a number of ways, and it may be tempting to think of it as a tool that can help you conduct your automated end-to-end tests at the end of development cycles more quickly. Yes, it can do that, but its biggest strength lies elsewhere. Visual AI allows you to shift left and begin to conduct testing “in-sprint” as part of an Agile development cycle.

Testing “in-sprint” means conducting visual validation alongside data validation and gaining complete test coverage of UI changes and visual regressions at every check-in. Bottlenecks are removed and releases are both faster and contain fewer errors, delivering an uncompromised user experience without jeopardizing your brand.

Teams that incorporate automated visual testing throughout their development process simply release faster and higher quality software.

How Can You Use Visual AI Today?

Wondering how you can move your organization or your team over to the left side of the bar charts above? Fortunately, it’s not hard to get started, and this chapter from Sogeti is an excellent place to begin. Keep reading to learn more about:

• When you should visually test your UI (and why it’s important)
• How to automate UI validation with Visual AI (including the three biggest challenges and how to overcome them)
• Different Visual AI comparison algorithms
• How Visual AI significantly reduces test creation and maintenance time while increasing coverage
• Streamlining analysis of test results and root cause analysis
• Using Visual AI for end-to-end validation
• Validation at check-in with Visual AI
• How Visual AI is revolutionizing cross browser testing

Deliver Quality Code Faster with Visual AI

Ultimately, what we’re all looking for is to be able to deliver quality code faster, even as complexity grows. Keeping up with the growing pace of change can feel daunting when you’re relying on traditional test automation that only scales linearly with the resources allocated – AI-powered automation is the only way to scale your team’s productivity at the pace today’s software development demands.

Applitools’ Visual AI integrates into your existing test automation practise and is already being used by the world’s leading top companies to greatly accelerate their ability to deliver innovation to their clients, customers and partners, while protecting their brand and ensuring digital initiatives have the right business outcomes. And it’s only getting better. Visual AI continues to progress as it advances the industry towards a future of truly Autonomous Testing, when the collaboration between humans and AI will change. Today, we’re focused on an AI that can handle repetitive/mundane tasks to free up humans for more creative/complex tasks, but we see a future where Visual AI will be able to handle all testing activities, and the role of humans will shift to training the AI and then reviewing the results.

Check out the full chapter, “Shorten release cycles with Visual AI,” below.

The post How Visual AI Accelerates Release Velocity appeared first on Automated Visual Testing | Applitools.

AI is needed to meet the scale and complexities of modern software delivery. According to the EMA report, traditional test automation tools will continue to struggle to keep up.

A growing challenge for organizations reliant on software (in other words, just about every organization today) is the ever-rising scale and speed of software delivery. Software is growing more complex, users are demanding more from their experiences, and release cycles are getting shorter and shorter. All of this puts an enormous strain on the testing teams charged with ensuring that applications are error-free and delivering the desired user experience.

AI is one technology that can ease this burden on today’s testers, according to EMA Research, which has just released a research paper on the topic. The need to create better software, faster, has never been greater.

“Business’s ability to accelerate the delivery of customer value through software innovation, at lower cost, has become critical for achieving competitive advantages,” said Torsten Volk, Enterprise Management Associates Managing Research Director.

Software Testing Complexity is Increasing

The report highlights a number of data points showing the increasing complexity of testing environments. The number of test automation-related questions posted to StackOverflow has nearly doubled over the past year. Smartphones continue to proliferate at a very high rate (30% CAGR since 2017 for Android alone), adding yet more configurations that need to be tested. Apps residing in a growing number of cloud services has risen 225% since 2015, compounding the complexity of software delivery.

In an article on the topic, VentureBeat recently highlighted another report on enterprise software development by Gatepoint Research which emphasizes some of the same struggles. According to that report, 77% of respondents said that they experience setbacks in releasing new software. A smaller but still high 34% said that fixing bugs takes anywhere from days to months.

Overall, EMA found that the increase in the complexity of technology, combined with faster release cycles and the daily tasks that already exists for test engineers, combines to yield an exponential increase in testing effort required. It doesn’t help that, as EMA puts it, “test automation frameworks typically rely on a jungle of test scripts written in different languages, using different sets of runtime parameters, and lacking consistent compliance test capabilities.”

AI is Critical to Modern Software Testing According to EMA

The research by Torsten Volk and his team makes clear in no uncertain terms that “you cannot scale automated testing without AI.”  The paper outlines how the latest AI solutions help in five key categories today, and digs into how AI can help address six of the biggest test automation pain points.

“AI-based test automation technologies can deliver real ROI today,” said Volk, “and come with the potential of addressing, and ultimately eliminating, today’s critical automation bottlenecks that stifle modern software delivery.”

EMA’s research discusses several essential AI capabilities that can combined and customized according to an organization’s requirements. Visual inspection (with Visual AI) was rated as having the highest overall impact of these capabilities, both today and in the future:

Smart crawling, self-healing, anomaly detection, and coverage detection each are point solutions that help organizations lower their risk of blind spots while decreasing human workload. Visual inspection goes further compared to these point solutions by aiming to understand application workflows and business requirements.

-Disrupting the Economics of Software Testing Through AI

To learn more, download the complementary report, “Disrupting the Economics of Testing Through AI.”

The post How AI Can Help Address Modern Software Testing appeared first on Automated Visual Testing | Applitools.

Code completion is nothing new. Tools like IntelliSense have allowed developers to become more productive by attempting to automatically complete the name of a function or statement they’re in the middle of writing, but the tools available only have a certain level of actual “intelligence” available.

With GitHub’s limited release of Copilot, they’re taking advantage of the powers of AI to boost developer productivity by not only trying to complete a function’s name, but by trying to suggest the entire function itself!

What is GitHub Copilot?

GitHub Copilot is a new tool from GitHub that provides realtime code suggestions when working inside of VS Code as an extension.

It’s powered by OpenAI, trained on billions of lines of public code, courtesy of projects hosted on GitHub itself, giving it the ability to have a wide variety of authors and languages to base those suggestions off of.

How does it work?

Copilot will typically give suggestions as you type in any scenario, but the two compelling examples that GitHub provides is being able to create a function based off of a comment or based off of the name of the function.

For instance, if I were to write:

// returns a sorted array of objects based on date

Copilot will get started with a new function, then once accepted, continue to write that function.

Alternatively, I can supply the function name like:

function sortArrayByKey

Where Copilot could then write that entire function for me!

If I don’t like the first answer it gives, I also have the option to look through alternatives, including pulling up all the answers Copilot thing could be helpful in this context.

What other kinds of things can it do?

There aren’t really any hard limitations beyond the basic gist of the tool being a way to suggest code. That just means you’ll be able to get a suggestion for finishing a function, but not necessarily for scaffolding a project with new files and directories.

While it can suggest really anything like an entire React component, it seems to really excel at utility functions that have a clear goal such as custom functions to sort an array like above or filtering data. Building components can be pretty specific and not include dependencies, where those functions often only require a specific input and output.

Copilot can even read context from the active file. If you’re trying to sort or filter data based on what exists in that file, it will try to use that context in its suggestion.

Can it write tests?

Writing functions is one thing, but can Copliot produce real, meaningful tests? Well, it depends.

Context is an important factor with tests. If you’re writing a unit test with Jest to harden business logic or a Cypress test to walk through your UI, that information is typically scattered in other files or not even available until it renders in a browser. Copilot can write a test based on similar situations, but it wouldn’t “just work”.

That said, when using testing frameworks like Applitools, you can get visual testing coverage with code that doesn’t need to be super specific to your application, only requiring it to load in the browser, where Applitools steps in and does the heavy lifting.

Part of Copilot’s wide net of training data includes a variety of Applitools examples, meaning we can easily write our new Eyes checks right in our existing testing frameworks.

Is Copilot the future of development?

The release of Copilot and tools like it bring a new era in what the software we’re using can do for us in day-to-day development. By leveraging AI and machine learning, we’re able to train computers to do the hard lifting and mundane tasks while we focus on solving the real problems, not how to fight with our text editor.

For a lot of developers, getting this information is a typical Google search with a look at a Stack Overflow answer. Even if only using it for common functions, we’re saving time and mental resources not having to look those things up ourselves.

But Copilot is still just another tool in our belt. It shows us what’s possible, but like all other tools, it’s up to the developer to ultimately take that knowledge and use it to its potential.

The post Using GitHub Copilot to Automate Tests appeared first on Automated Visual Testing | Applitools.

Applitools was invited to share our expertise in applying AI to quality engineering, and we’re honored to be co-authors of this comprehensive report by Sogeti.

Sogeti has just released the first section of their State of AI applied to Quality Engineering 2021-22 report, including two chapters co-authored by Applitools. The report is a detailed examination of the current state of artificial intelligence in the field of quality engineering. It centers around a key question – how can AI make our quality validation smarter? In the words of the executive introduction:

This report aims to assist you in understanding the potential of AI and how it can help improve the quality, velocity, and efficiency of your quality engineering activities.

As one of the pioneers in the application of AI to quality engineering through Visual AI, we were honored to be asked to participate in this report and share our expertise. We co-authored several chapters, including two that have been released today in the first section.

Chapter 1: Quality Engineering and recent trends

In this chapter, you’ll get an overview of the business and technical environment which has led us to where we are today and the current need for assistance from AI. It discusses the shortcomings of traditional testing practices and the emergence of modern quality engineering. What does a successful Quality Engineer do today? What are the challenges faced? What is the future of quality engineering, and what role could AI play in that? Check out this opening chapter for a great introduction into the topic of AI in QE. 

Chapter 2: Getting started with AI

This chapter digs a little deeper into how you can get started in your journey with AI. Moshe starts by relating a personal story about a customer service experience that left him frustrated. How can we use AI to eliminate waste from our days and spend more time on quality engineering and address issues before they impact end users? The chapter goes on to cover the difference between routine and error-prone tasks and opens up the discussion of how we can optimize each type. You’ll also get some great info on how to define AI, understand possible use cases, and thoroughly research your options. Head over to the second chapter to read more.

In chapter 3 and chapter 4, you can explore further with technical deep dives into machine learning and deep learning. 

Check out the Full Report

Sogeti has put together a strong report on this important topic and we’re excited to share the opening section with you today. Starting in September, you can expect to find new sections released bi-weekly, including another chapter from Applitools that will be out in the coming months. To learn more, check out the full “State of AI applied to Quality Engineering 2021-22” report. 

The post The “State of AI applied to Quality Engineering 2021-2022” Report Released appeared first on Automated Visual Testing | Applitools.

Artificial Intelligence and Machine Learning (AI/ML) have seen application in a variety of fields, including automation of QA tasks. But what are they exactly? What distinguishes different instances and applications of AI, for example? What are the horizons of these technologies in the field of QA?

The promise of AI/ML must be understood correctly to be harnessed appropriately. As with any buzzword, many technologies and products are offered under the guise of AI/ML without satisfying the definition. The industry is reforming itself around the promise that AI/ML holds often without a clear understanding of the technical limitations that give the promise its boundaries.

In this webinar, test automation guru Jonathan Lipps gives a detailed overview of the concepts that underpin AI/ML, and discuss their ramifications for the work of QA automation.

In addition to a discussion of AI/ML in general, Jonathan looks at examples from the QA industry. These examples will help give attendees the basic understanding required to cut through the marketing language. so we can clearly evaluate AI/ML solutions, and calibrate expectations about the benefit of AI/ML in QA, both as it stands today and in the future.

Jonathan’s slide deck:

Full webinar recording:

Additional resources:

• Jonathan Lipps’s course on Test Automation University: “Automated Visual Testing with Appium” — take Jonathan’s course on visual testing for mobile apps w/ Appium, exclusively on TestAutomationU!
• AI, Please Test My App! — webinar w/ Gil Tayar
• AppiumPro — Jonathan’s weekly newsletter on all things Appium and mobile test automation; you can subscribe here
• “Modern Functional Test Automation Through Visual AI” — Take this free online course — only on Test Automation University!
• Angie Jones Presents: “Automated Visual Testing: A Fast Path To Test Automation Success” — ace automated visual testing with this free 9-chapter online course by Angie Jones on Test Automation University
• 7-year Old Girl vs. Visual AI — blog post
• Testing iFrames with Visual AI — blog post by Michael Battat
• Data-driven Testing w/ Visual AI – blog post by Michael Battat
• Is AI Required for Successful Digital Transformation? — blog post by Al Sargent
• Start visual testing today — open your free Applitools account here

— HAPPY TESTING —

The post Myth vs Reality: Understanding AI/ML for QA Automation – webinar w/ Expert Jonathan Lipps appeared first on Automated Visual Testing | Applitools.