Mobile app testing tools embrace automation, open source

Today's end users expect more from mobile apps than ever, and mobile app testing tools can ensure that an app is as close to perfection as possible upon release.

Enterprise apps in particular are a good fit for mobile app testing because any failed feature could detract significant value from the business. And today's mobile app testing tools eliminate the need to search for real devices and testing subjects.

Sauce Labs Inc., based in San Francisco, provides automated mobile app testing tools. The company this week announced support for Google Android and Apple iOS native test automation frameworks such as XcuiTest and Espresso. In this Q&A, Steven Hazel, co-founder and CTO of Sauce Labs, discusses best practices and trends around mobile app testing tools.

Why is it important for developers to use a mixture of mobile app testing tools, such as real devices, emulators and simulators?

Steven Hazel: A functional test suite typically includes a lot of tests; you're trying to check out the functionality of every feature of your app. But in a modern, continuous integration and delivery workflow, you want to get through those tests in a short amount of time so you can get feedback to your developers if there's some kind of problem.

You want to ... run many tests at the same time. That's most cost-effective on emulators and simulators because it's easy to ramp up tons of them in a short amount of time, and they can be very reliably reset to a known good state.

But emulators and simulators are not perfect reproductions of real devices.

But emulators and simulators are not perfect reproductions of real devices. They lack some of the features of real devices, and they can't touch some of the unique aspects, like geolocation, access to [the] camera and anything that involves hardware.

A mixture of the two is a good way to get through a large test suite in a short amount of time very reliably and increase the fidelity of your testing -- especially on new features or spot-checking older features that might be affected by new changes.

What are some metrics that developers should look at to maintain mobile app health?

Steven Hazel

Hazel: Looking at things like application performance monitoring is a best practice. You definitely want to track the behavior of your app when it's used in production.

As people move toward microservices, there is an increasing need for distributed tracing, so things based on the OpenTracing spec are great ways to wrangle the complexity of microservices and be able to trace interactions across your app.

What are the challenges that microservices bring to mobile app development and testing?

Hazel: Microservices split up the functionality of the app and allow teams to work much more independently. There's much less focus in that kind of environment on end-to-end testing across all the services, although you still need some of that.

With a microservices architecture, it's helpful to try to isolate your microservice that you're working on and virtualize all of the services that it works with. The end-to-end testing in that kind of environment can come a little bit later if you have a higher degree of confidence based on service virtualization, like contract testing that you were doing in the earlier phase.

Once you're in production, [microservices] affect how you do metrics and performance monitoring. You no longer have a single monolithic app where you can ask the app what it's doing in handling particular requests; you have to assemble that picture from what different microservices did.

What are some common mistakes that developers make when testing?

Hazel: Outsourcing your testing, putting your testing efforts off and saying, 'We just really need to get the app built, we'll hire some quality assurance [QA] folks and they'll test it, but we don't have time to work with them right now.' It's very shortsighted.

The teams that wind up in a good position are thinking about testability from the beginning. They have developers and QA experts involved in creating those tests. They have an automated test suite to run every time they push a change to their app, and they just add onto that test suite over time. They try to make that test suite reliable and build the app so that it's easy to create tests.

What are some of the trends you're seeing around mobile app testing tools?

Hazel: Many dev tools now are open source. Git is the most popular version control system now. There were many commercial version control systems in the past, but having Git be open source opens up some flexibility and options for developers. There are commercial services built around that, like GitLab and GitHub.

There's kind of a split there that becomes pretty familiar where the tool itself is open source. That gives dev teams a lot of flexibility to try it out; explore different ways that they might want to use it that are unique and novel and innovative.

But when there's an established need for a service, that [service] needs to be high quality and work with those tools. Then, dev teams still get the flexibility of open source, but they also get the scalability and reliability of commercial SaaS services.