Breakdown of 4 common mobile testing platform options
Mobile app testers, here is how four mobile testing platform options compare. Also, learn when testing frameworks, simulators and emulators become relevant factors.
Mobile testers have a growing number of devices and OSes being thrown at them. Plus, with new tools, frameworks and -- most notably -- platforms available, it's important to make use of testing tools that cover the most important device and OS combinations an application uses.
The market offers many different testing platform options, each with their pluses and minuses. Below, we compare the following:
Sauce Labs
Perfecto
BrowserStack
AWS Device Farm
Choosing a mobile testing framework
When choosing a testing platform, be sure to have a testing framework that supports that platform -- and vice versa.
Once the mobile testers choose whether they will do their QA via emulators, simulators, real device testing or some combination, the team must choose its testing framework. A testing framework is the code library QA professionals use to write and execute the tests.
A testing framework is the code library QA professionals use to write and execute the tests.
The testing framework provides functions to control the test device and make assertions about its state to facilitate testing. Some options are the following:
Appium
Calabash
JUnit
Espresso
Robotium
XCTest
Selenium
The framework a team choses depends on the OS the application runs on -- Android, iOS or Windows -- and what features the tests assess. Not all frameworks offer the same testing capabilities. And not all frameworks work with every language; teams may have to choose accordingly. For example, Appium -- a testing framework option all platforms mentioned below support -- supports Ruby, Python, Java, JavaScript, PHP and C#.
Simulators, emulators or the real McCoy
Let's clarify some terminology. Mobile testing can take place with either an emulator, simulator or a real device.
Simulators. Simulators are software programs that serve to replicate only the software of a mobile device. A simulator that mobile testers commonly use, for instance, is Apple's iOS Simulator. With a simulator, QA professionals are unable to test as much as they can with an emulator.
Emulators. Emulators are software programs that serve to replicate the software and hardware of a mobile device. For example, QA professionals often test Android software with an emulator from Google. Since emulators mimic both the software and hardware, such programs enable mobile testers to assess Bluetooth connectivity or how an application runs in different battery states.
Real device testing. With a real device, the mobile tester runs the application on an actual device; to run tests, an automated framework may control it. This approach can involve either a physical device the tester connects to a computer or one they access over the internet. Real device testing enables full testing of an application, without any of the holdbacks of simulators or emulators.
Selecting a mobile testing platform
Testing platforms are SaaS offerings where QA professionals run tests using the platform's computing power.
Testing platforms are SaaS offerings where QA professionals run tests using the platform's computing power.
Use of these platforms frees up testers from the responsibility of configuring and maintaining simulators and emulators, as well as the orchestration that would be needed to run the tests. A testing platform also enables QA professionals to execute many tests at once, speeding up the test suite. Most of these platforms use a monthly pricing model, based on number of tests run, length of the test in minutes and the number of devices used.
Sauce Labs. This platform is a common option that offers many different device options. Sauce Labs supports mobile testing with emulators and simulators, as well as real device testing in the cloud. Its offerings cover both Android and iOS, supporting the latest device and OS versions from each. I've enjoyed using Sauce Labs on mobile projects in the past and found its product to be easy to use and quick to learn. For mobile testing frameworks, Sauce Labs supports Appium, Espresso, XCUITest and Robotium.
Perfecto Mobile. This tool is like Sauce Labs in what it offers. It supports emulators, simulators and real devices through the cloud. Perfecto supports the following testing frameworks: Selenium, Appium, XCUITest, Espresso and Quantum.
BrowserStack. Another solid mobile testing platform with simulators, emulators and real device testing. The tool supports many different Android and iOS devices. Its platform covers the Appium, Espresso, XCUITest and EarlGrey testing frameworks.
AWS Device Farm. This service is a newer option; it offers only real device testing and at a different pricing model than most of the other options. AWS Device Farm charges $0.17 per device minute or unlimited testing at $250 per month per device slot. The devices supported are not as numerous as other platforms, but popular Android and iOS devices are covered. AWS Device Farm supports the Appium, Calabash, JUnit, Espresso, Robotium, UI Automator and XCTest UI frameworks.
Ask about a free trial, which enables you to be sure your testing framework is well supported and the CI/CD integrations work well with a project's pipeline. All of the above platforms offer integrations with popular CI/CD systems, such as Jenkins, CircleCI and GitLab CI/CD.
Regardless of which platform a team chooses, carefully weigh the positives and negatives among emulators, simulators and real devices. A real device may be required to properly test some features, but most tests can be run speedily on an emulator or simulator.
Dig Deeper on Software testing tools and techniques
