Andy Knight (moderator):

So our first question comes from Jonathan Nathan.

Can Playwright or Cypress handle multiple browser tabs? What do engineers in these tools do for Azure authentication or target new links?

Gleb Bahmutov (Cypress):

[It’s a] very specific kind of interesting question, right? Multiple tabs and Azure authentication. You must have both. I think Playwright is [a] better tool, because it supports multiple browser tabs control right out of the box. So I would go with that.

Carter Capocaccia (Cypress):

Can I share a hack for this real quick?

Andy Knight (moderator):

Sure.

Carter Capocaccia (Cypress):

So I’m going to share the way that you don’t have to deal with multiple browser tabs in Cypress. And that’s just, you change the DOM from a target blank to a target self and then it just opens in the same window. So if you have to use Cypress [and] you need to get around multiple browser tabs, you can do DOM manipulation with Cypress.

Andy Knight (moderator):

Man, it sounds kind of hacky, though.

Carter Capocaccia (Cypress):

No, I mean it’s absolutely right. Yeah, it’s hacky.

Andy Knight (moderator):

So I got a question then. Is that like a feature request anyone has lifted up for Cypress? Or is that just because of the way the Cypress app itself is designed that that’s just a non-starter?

Gleb Bahmutov (Cypress):

It’s an open issue. Cypress team says it’s not a priority. Because you think about two tabs, right? Two tabs that present communication between one user and another of a backend. So you always want to kind of stop that and control it so you don’t have to actually have two windows. You want to control one window and communicate with API calls on the back. At least that’s the Cypress team opinion. So we might not see it any time soon.

Andy Knight (moderator):

Yeah. I know in Playwright like, Gleb, like you said, it is really easy and nice because you have a browser instance. Then in that instance, you have multiple browser contexts and then, from each browser context, you’re going to have multiple pages. So yeah, I love it. Love it.

Tally Barak (Playwright):

And the better thing is that you can have one of them, like in a mobile size or emulating a mobile device and the other one in web. So if you want to run a test of like cutting between two different users, each one is incognito and is a complete browser context. They don’t share their local storage or anything. So basically, you can run any test that you want on the thing. And because of the browser, it also works really fast. Because it’s not really launching a whole browser. It’s just launching a browser context, which is much, much faster than launching the whole browser.

Andy Knight (moderator):

Awesome. Alright, let’s move on to our next question here. This is from Sundakar.

Many of the customers I worked with are preferring open source. And do you think Applitools will have its place in this open source market?

Can I answer this one? Because I work for Applitools.

For this one, I think absolutely yes. I mean all of Applitools Eyes SDKs are open source. What Applitools provides is not only just the mechanism for visual testing, but also the platforms. We work with all the open source tools, frameworks, you name it. So absolutely, I would say there’s a place here. Let me move on to the next question.

Gleb Bahmutov (Cypress):

Andy, before you move on, can I add? So my computer science PhD is in computer vision image processing. So it’s all about comparing new images, teaching them, and so on. I would not run my own visual testing service, right? My goal is to make sure I’m testing [the] web application. Keeping images, comparing them, showing the diffs, updating it. It’s such a hassle, it’s not worth it, my time. Just pay for a service like Applitools and move on with your life.

Andy Knight (moderator):

Awesome. Thank you. Okay. Let me pull the next question here. This is from Daniel.

I heard a lot that Playwright is still a new framework with a small community even when it was released in January of 2020 but never heard that about WebdriverIO. As far as I know, Playwright is older.

I don’t think that is true. I’d have to double check.

Tally Barak (Playwright):

No, I don’t think [so].

Andy Knight (moderator):

Is Playwright still considered new?

Tally Barak (Playwright):

It’s newer than the others. But it’s growing really fast. I mean, because I’m the [Playwright] OG, I remember the time when I would mention Playwright and no one had any idea what I’m talking about. It was very, very new. This is not the case anymore. I mean, there’s still, of course, people don’t really hear about it, but the community has grown a lot. I think [it has] over 40,000 stars on GitHub. The Slack channel has almost 5,000 participants or something. So the community is growing, Playwright is growing really, really nicely. And you’re welcome to join.

Andy Knight (moderator):

Okay, here’s another question from Ryan Barnes.

Do y’all integrate automated tests with a test case management tool? If so, which ones?

Gleb Bahmutov (Cypress):

Yes. TestRail.

Andy Knight (moderator):

TestRail. Okay.

Gleb Bahmutov (Cypress):

Because we are not the only testing tool, right? Across organizations, where our teams, our tools, and manual testing. So we need a central testing portal.

Tally Barak (Playwright):

No, we are not perfect. And we should. Any good ideas are welcome.

Carter Capocaccia (Cypress):

So we don’t have a formalized test manager tool. But if anybody has ever used any kind of Atlassian tooling – there’s, you know, JIRA out there has the idea of a test set ticket inside of the test set or individual test. You can define user flows inside of there. So I guess you can consider that a test management tool. It’s a little bit less featured than something like TestRail. Actually, it’s a lot less featured than something like TestRail. But you know, that’s how we stay organized. So we basically tie our tests to a ticket. That’s how we can manage, you know, well what is this ticket test? What is it supposed to be testing? Where is our source of truth?

Andy Knight (moderator):

I guess I could launch a somewhat controversial question here, but I’ll do it rhetorically not to answer. But if you have a test automation solution, do you really need to have it export results to a test case management tool? Or can you just use the reports it gives you? We’ll leave that for offline. So the next one on the list here is from Sindhuja.

We are trying to convert our test scripts from Protractor.

Okay. Wow, that’s a blast from the past.

We are doing [a] proof of concept and WebdriverIO and we have issues with running in Firefox and WebdriverIO. Is there any notes in WebdriverIO with cross browsers?

Jose, got any insights for us here?

Jose Morales (WebdriverIO):

Yeah, absolutely. So one thing that I really love about WebdriverIO is the easy configuration. So, when you create a project in WebdriverIO, you have a JSON file where you put all the configuration about capability services, what browser you want to use. And you can easily add your own configuration. It could be, for example, if you want to run in Firefox or in Edge or you want to run on Source Labs, you have several options.
So it is really easy to integrate configuration for Firefox. You only need to specify the browser in the capability section along with the version and special features like size view. If you want to know how to do that, it’s very easy. You can go to my home page. And there [are] examples where you can build something from scratch and you can see easily where to add that particular configuration. And I’m going to share with you some repositories in GitHub where you can see some examples [of] how to do it.

Andy Knight (moderator):

Thank you so much, Jose. Oh, here we go.

Which framework would be the best platform to test both Android and iOS apps?

I know most of us [are] focused on web UI, so here’s a curveball: mobile.

Gleb Bahmutov (Cypress):

I can say that at Mercari US, we picked Detox after using Appium for a long time. But for new React Native projects, we went with Detox.

Carter Capocaccia (Cypress):

Yeah, Detox is the only one that I’ve ever used it for as well. And it was really, really good. I found no reason to switch. I think, Gleb, can you correct me if I’m wrong on this? I think Detox was originally made by Wix? Is it, Wix, the company?

Gleb Bahmutov (Cypress):

That’s correct. Yes.

Carter Capocaccia (Cypress):

Yes, so Wix used to make Detox. I think it’s still maintained by them, but it was like an in-house tool  they open sourced, and now it’s really good.

Andy Knight (moderator):

Awesome. Cool. I hadn’t heard. I’ll have to look it up. Alrighty, well I think that’s about all the time we have for question and answer today. I want to say thank you to everyone for attending. Thank you to all of our speakers here on the panel.

Andy Knight (moderator):

So I’m going to ask another popcorn question here. And this one again comes with an audience poll. So if you’re in the audience, hop over to the polls and take a look here. Have you ever considered switching your test tool or framework? And if so, why? If so, why not?

Carter Capocaccia (Cypress):

Yeah, we considered it and did it. You know, I think you have to acknowledge it at a certain point that maybe what you’re doing isn’t working or the tool doesn’t fit your application or your architecture. And it’s a really hard decision to make, right? Because typically, by the time you get to that point, you’ve invested tens, hundreds, if not thousands of hours into a framework or an architecture. And making that switch is not an easy task.

Andy Knight (moderator):

What did you switch from and to?

Carter Capocaccia (Cypress):

Yeah, so [Selenium] WebDriver to Cypress. And you can already kind of see that architecturally they’re very, very different. But yeah, it was not an easy task, not an easy undertaking, but I think we had to admit to ourselves that WebDriver did not fit our needs. It didn’t operate in the way we wanted it to operate. At least for us, it was set up in a way that did not match our development cycles, did not match our UI devs’ patterns and processes.

So when we talk about, you know, kind of making a team of test engineers have additional help, well, one way to do that is [to align] your test automation practices with your UI dev practices. And so if we can have a JavaScript test framework that runs in a web app, that acts like a web app, that feels like a web app, well, we’ve now just added an entire team of UI developers that we’ve kind of made them testers, but we haven’t told them that yet. That’s kind of how that works. You adopt these people by just aligning with their tooling and all of a sudden they realize like, hey, I can do this too.

Andy Knight (moderator):

Mm-hmm. So I want to ask another question. Since you’re saying you used Selenium WebDriver before, in what language did you automate your test? Was that in JavaScript, just like first test, or was that in Java or Python or something else?

Carter Capocaccia (Cypress):

Yep, so it’s JavaScript. But when we look [at the] architecture of the app, right? So the architecture of WebDriver is very different than Cypress. [With] Cypress, you write a JavaScript function just like you write any other function. And most of the time, it’s going to operate just like you expect it to. I think that’s where, instead of them having to learn a WebDriver framework and what the execution order of things was and how the configurations were, Cypress is just pretty apparent in how it functions. When you look at like, well, how is the Cypress test executed? It uses Mocha under the hood as the test runner. So that’s just functions that are just kind of inside of describe and context and it blocks. So I think anybody that doesn’t know anything about testing but is a JavaScript developer can hop right into Cypress, look at it, and say, yeah, I think I know what’s going on here. And probably within the first 5 to 10 minutes, write an assertion or write a selector.

Andy Knight (moderator):

Awesome. That’s really cool. And so was that something that you and your team or org did recently? Was this like a couple years ago that you made this transition?

Carter Capocaccia (Cypress):

You know, I think I’d be lying to you if I told you the transition ever ends, right? I think it starts and you get to a point where you’ve scaled and you’ve gotten to where you’ve written a ton of tests on it, and then that’s great. But of course, it’s kind of the bell curve of testing. Like it starts really, really slow because you’re moving off the old framework, then you start to scale really quickly. You peak, and then at that point, you have these kind of outliers on the other side of the bell that are just running legacy applications or they just don’t want to move off of it. Or for whatever reason, they can’t move off of it. Let’s say they’re running like Node 10 still. It’s like okay, well, maybe we don’t want to invest the time there to move them off of it.

Yeah. So it’s a long journey to do a task like this. So it’s never done. By the time we’re finished, you know, there’s gonna be a new testing tool or Cypress will have a new version or something else will be new in there. You know, pipelines will [be through] some AI network at this point. Who knows? But I’m sure we’ll have to start the next task.

Andy Knight (moderator):

Sure. So how about others in our group of panelist speakers here? Have y’all ever transitioned from one framework or tool to another? Or in your current project, did you establish it and you were just super happy with what you chose? And tell us the reasons why.

Tally Barak (Playwright):

Yes. So, yeah, we also switched from WebdriverIO as I said at the beginning. We switched to Playwright, obviously. It was not that painful, to be honest, maybe because we kept the Cucumber side. So we had [fewer] scenarios than what we have today. So it was less to move. I think it went over a period [of] a few months. Unlike you, Carter, we actually moved completely away from WebdriverIO. There is no WebdriverIO, everything is with Playwright. But again, probably a smaller organization, so we were able to do that. And are we happy? I mean, [we are] thankful every single day for this switch.

Gleb Bahmutov (Cypress):

I can comment. At Mercari, we did switch from WebDriver in Python and mabl. So both were because the maintenance of tests and debugging of failed tests was [a] very hard problem, and everyone agreed that there is a problem. The transition was, again, faster than we expected, just because the previous tests did not work very well. So at some point we were like, well, we have 50 failing daily tests, right? It’s not like we can keep them. It’s not a sunken cost. It’s a cost of not working. So we have to recreate the test. But it was not that bad. So for us, the cost of debugging and maintaining the previous tool just became overwhelming, and we decided, let’s just do something else.

Jose Morales (WebdriverIO):

In my experience, we were using Selenium with JavaScript as a previous framework, and then we moved to WebdriverIO with JavaScript and Applitools. Because in Domino’s, we have different kind of products. We have another product that is called Next Generation of the stores. We’re using a reactive application there. This is also a website application. And we’re evaluating other tools for automation such as UiPath. And one thing that I like about UiPath is [it’s] a low-coding solution. That means you don’t need to really be a very proficient developer in order to create the scenarios or automate with UiPath, because it only [requires] drags and drops [to] create the scenarios. And it’s very easy to automate. And in the top of that, with UiPath, it’s a very enterprise solution.

We have different kinds of components like an orchestrator, we have a test manager, we have integration with Jenkins. We have robots that execute our test cases. So it’s an enterprise solution that we implemented for that particular project. And we have the same results as WebdriverIO with Applitools. We’re happy with the two solutions – the two frameworks – and we’re considering to remove WebdriverIO and Applitools and use UiPath for the main website for Domino’s. So I think in the future, the low-coding solution for automation is a way to go in the future. So definitely UiPath is a good candidate for us for our automation in the website.

Andy Knight (moderator):

How about you, Steve? I know you’re right now on a Selenium-based test project. Have you and your team thought about either moving to something else, or do you feel pretty good about the whole Selenium-ness?

Steve Hernandez (Selenium):

I think clearly there appears to be major speed-ups that you get from the JavaScript-based solutions. And I know you’ve evangelized a little bit about Playwright and then I’m hearing good things from others. I mean, we have a mature solution. We have 98% pass rates. You know, some of the limitation is more what our application can handle when we blast it with tests. I don’t know if your developers after each local build are tests running locally using some of these JavaScripts solutions. But I think one of the things that is appealing is the speed. And with Boa Constrictor 3, the roadmap is to – well, it has been modularized. So you can bring your own Rusty API client or potentially swap in Selenium and swap in something like a module for Playwright. So we would get that speed up but keep the same API that we’re using. So we’re toying with it, but if we can pull that off in Boa Constrictor, that would take care of that for us.

Andy Knight (moderator):

Yeah, man, definitely. So, I mean, it sounds like a big trend from a lot of folks has been, you know, step away from things like Selenium WebDriver to [a] more modern framework. You know, because Selenium WebDriver is pretty much just a low-level browser automator, which is still great. It’s still fine. It’s still awesome. But I mean, one of the things I personally love about Playwright – as well as things like Cypress – is that it gives you that nice feature richness around it to not only help you run your tests, but also help develop your tests.

Andy Knight (moderator):

Alrighty. So I mean, this, our conversation is now kind of going in so many good directions. This is awesome. We keep talking about integrations and how our frameworks and our tools can connect with other things. So I guess my next question I would like to ask as a big popcorn question would be: what are some of your favorite integrations with your test projects? I’m talking about things like CI hooks, special reports that y’all have, maybe things for visual testing, accessibility testing.

And not only is this a question for the panel speakers, but also for the audience. We have another Slido word cloud, so head over to the Slido tab. We’d love to hear what your favorite integrations are for your test projects.

But I’d like to now direct this question back to our panel and ask them to pick one. Tell me why you love it so much. Tell me how y’all use it and what value y’all get out of it.

Carter Capocaccia (Cypress):

Okay, I’ll hop in here. I think for me with Cypress, it’s just under the hood, it’s just JavaScript, right? So most anywhere is able to hook into that immediately. So when we talk about integrations, let’s just use GitHub for example here. If you wanted a GitHub Actions pipeline, you can just go ahead and start doing that immediately. And Cypress out of the box will provide you a Docker image to go ahead and use right inside of your GitHub Actions pipeline.

So for me, I guess I’m not going to focus on any one specific tool for integration here. I think the way that Cypress kind of makes itself agnostic to all external tools is what makes it really, really nice. You know, it doesn’t say, hey, you’ve gotta use us with this particular, like an Azure pipeline or any kind of other DevOps tool. It says, hey, under the hood, all we are is a JavaScript framework. If you want to execute our JavaScript inside of some kind of Docker image, create your own Docker image. Gleb knows all about this. He publishes a new Docker image like every day it seems.

And we have, you know, the ability to take all that, take the results, and the base of what comes out of Cypress is just JSON. We’ll shoot that over to an open-source dashboard. So there’s Report Portal out there if you wanted to take those results, and then put them into a portal. I see on here [the poll results] Allure reports. Well, okay, so you take it, you just put it into any kind of integration there.
So I think for me personally – I actually wrote a whole blog post about this where I was able to take Vercel, GitHub Actions, and Cypress and set up a CI/CD pipeline completely for free with automated deployments, preview environments, automated testing, the whole nine yards. And so it made it to where for my personal little website, all I’ve got to do is submit a pull request. It kicks off all my tests. As soon as it’s done, it merges it, and then it’s immediately deployed with Vercel. So that was a really cool thing. So if I have to name a tool, I’m gonna say Vercel and GitHub Actions.

Gleb Bahmutov (Cypress):

Can I second that? GitHub Actions are amazing. Honestly, I went through the whole, you know, self-running Jenkins to TeamCity, Buildkite, Circle CI. GitHub Actions are the best. It’s amazing how much you can do with just JavaScript on UCI. And I was the one writing all integrations for the Cypress team at the time.

I want to bring one more integration. I’ve written it. It’s [an] integration between Cypress and LaunchDarkly feature flags environment, because during the test, sometimes you want to test an experiment behind a feature flag. And for us now, it’s an open-source plugin. It’s like this, here’s my user, here are the experiment values I want to enable. Let’s see this functionality, and it’s working.

Tally Barak (Playwright):

[My] answer is GitHub Actions as well. I don’t have that much experience derived with different tools. I used to work with Travis and a bit with CircleCI. But GitHub Actions is just answering everything we need – and Playwright obviously – because it just requires a node for the JavaScript version. You just install it with Docker, and you are good to go and do whatever you want with the outputs and so on. So yeah, GitHub actions and I’m actually interested in the other one that you mentioned about the feature flag. That’s worth looking into.

And of course, Cucumber. I also have a repository that is a starter report for working with Cucumber and Playwright. I’ll post it on the channel if anyone is interested to try it out. If you want to stick with the Gherkin and the BDD-style test. Otherwise, go with the Playwright test runner.

Andy Knight (moderator):

Tally, I want to plus one of both things you said there. First of all, everyone knows that I love BDD, I love Cucumber, I love Gherkin, so yes. But also, I want to go back to what everyone is harping about GitHub Actions. Y’all in the audience, if you haven’t tried GitHub Actions or used to yet, definitely check it out. It is very straightforward, low-to-no cost CI baked into your repo. I mean, like anytime I’ve done open-source projects or example projects, just putting a GitHub Actions to automatically run your tests from a trigger, oh, it’s beautiful. It can be a little intimidating at first because you’ve got that YAML structure you’ve got to learn. But I have found it very easy to pick up. I feel like personally – and speakers back me up on this – I never sat down and read a manual on how to write a GitHub Action. I just kind of looked at GitHub Actions and copied them and tweaked them, and it worked.

Tally Barak:

And then you went to the marketplace and looked for some.

Andy Knight (moderator):

Yes, yes, yes. You know, and so you can find things that are out there already. And I’ve done it not just with Cypress and Playwright, but literally like every single test framework out there that I’ve touched. You know, if it’s like a Selenium thing, if it’s a Playwright thing, a Cypress thing. Any kind of test unit integration, end-to-end, you name it. Like, it’s awesome. But anyway, I just wanted a plus one to both of those and now I’ll yield to the rest of the speakers’ favored integrations and why.

Steve Hernandez (Selenium):

Can I give a plug for LivingDoc? We use a lot of the SpecFlow stuff in our stack, as you saw, like the runner. There’s a great SpecFlow+ Runner report that gets generated and can show how each of the threads are performing in your test suite so you can look at performance. But one of the things just really simply is the Azure Log Analytics ingestion API. So right in your test pipeline, you can just throw up a PowerShell script or some console application and send that data that you would have in your CI tool over into something like log analytics. I suppose there’s an equivalent for Amazon. But then you can correlate, you know, your telemetry from your application along with test data and start to see where performance issues are introduced.

But I guess coming back to my plug for LivingDoc is, I feel in a lot of cases our automated tests are actually the only documentation for some parts of the application as far as how they’re supposed to work because you’ve captured the requirements in a test. So they’re kind of like living documentation and LivingDoc literally is that. And it’s something that you can share with the business and the developers to see what you have for coverage and it’s human readable. I think that’s really important.

Carter Capocaccia (Cypress):

Well, now with GPT3, everything is human readable.

Andy Knight (moderator):

So I wanted to ask about visual testing and specifically with Applitools. I saw and heard that some of y’all had used visual testing with Applitools in your projects. In line with talking about integrations that you love, how do y’all use Applitools if you use it, and what value do you find you get out of it? Go ahead.

Carter Capocaccia (Cypress):

Yes, I want to bring up kind of why Applitools is a different breed of product when it comes to visual testing, though. So you really have two flavors of visual testing. One is a bitmap comparison where you take a screenshot of an image and it takes it and basically breaks it down into computer readable code and it compares a known standard against the new version. If there’s a certain percentage with diversions between the old version and the new version, it flags it.You can kind of already tell, well, if all I know about is just basically one to zeros and comparing the results here, it’s not really all that intelligent. Because I could have a piece of the page, let’s say it’s a headline. The headline updates every five minutes. Well, I don’t have time to just continually update this baseline screenshot.

Well, what a tool like Applitools does is it actually takes your screenshots and puts them through computer vision. And with computer vision you can now provide a lot more instruction to what do you want to compare? What’s important to you? Do you want to ignore certain parts of the page? When I compare X to Y, do I really consider, let’s say, a carousel where the images are being delivered by a CMS and they’re being changed on a daily basis? Do I really care that yesterday it was a picture of let’s say somebody’s front porch and today it’s a picture of a tree? So there’s a lot more intelligence that’s built into a tool like Applitools than there is with standard bitmap comparison. I think it takes the right kind of customer for Applitools. And when they do have that right kind of customer, it’s a pretty huge game changer, considering that it allows you to make a lot more informed decisions about the health of the visual appearance of your application versus, like I said, a bitmap comparison.

Basically, if you want to compare the two, one is analog, the other is digital. The analog is the map comparison. The digital way is Applitools. If we’re being honest with ourselves, the Applitools way is probably the best and right way to do this. I just know that there are a lot of teams that are making smaller scale applications that can get away with assuming higher risk in using a tool like a bitmap comparison tool.

Jose Morales (WebdriverIO):

I would like to add that we have been using Applitools for six months or so, and it was a great addition to our framework. It was reducing a lot of code because we were coding a lot of assertions. So for example, if you go to the Domino’s website, you have the store locator, and then the store locator basically based on your zip code automatically populates the city region and the store closest to you. There is a lot of functionality there. And I remember that we were validating every single text fill, every single button, every single drop and menu for the content. In English and in Spanish and the different languages. So for us it was very helpful to use Applitools because with WebdriverIO, we can simulate the location from the customer and then move the store locator for that particular zip code and tell Applitools, hey, take a screenshot for this one.

And in one single shot, we are validating all the colors, buttons, text fields, content, and we were very careful to see what is worth it for this scenario, right? So in all the framework, what we are doing is okay, let’s test this particular functionality for this scenario. Let’s use WebdriverIO to move the customer until this particular checkpoint and then take a screenshot for that particular state in our application. And in one single shot, we can validate this in all the devices. In Chrome, Firefox, Edge. Pixel, iPhones. So with that particular action, and because we are reducing a lot of assertions, we are writing less code, it’s easy to maintain, it’s easy to extend, and our test cases are more accurate and smaller. And I think for us Applitools was a great addition and I fell in love with Applitools and currently we’re trying to incorporate the mobile validations, meaning iOS and Android as well.