momius - Fotolia
Compare web and mobile testing tools from AWS, Microsoft and Google
Compare and contrast the mobile testing tools from the big three cloud vendors -- AWS Device Farm, Google Firebase Test Lab and Microsoft Visual Studio App Center Test.
The hardest part of getting started with web and mobile testing in the cloud is picking a tool that works for testers and developers -- one that is both intuitive and easy to set up.
In the public cloud, you can virtually access a large variety of mobile devices and browsers for test and dev. For a fee, you'll get device access without having to manage, secure or house your own test lab.
And while there are numerous web and mobile testing tools available from a range of vendors, we'll stick to the services offered by the three major public cloud providers for the purposes of this piece. Let's review and compare the capabilities available on AWS Device Farm, Google Firebase Test Lab and Microsoft Visual Studio App Center Test to see which best meets your testing needs.
AWS Device Farm
AWS Device Farm provides accessible testing across a variety of established browsers and mobile devices. AWS handles provisioning and management of all devices, so users can focus on test structure and execution. AWS Device Farm includes these features:
- Built-in security for accessing devices and browsers, as well as your code.
- Concurrent testing across mobile devices and browsers.
- Test executions that provide videos and error logs to assist in analyzing issues.
- A tool that automates testing without having to script or maintain code.
- Ability to test mobile apps on native or hybrid web code bases.
- Available browsers that integrate with Selenium, in case you already use scripted Selenium test automation.
- Integration with Appium, Calabash, Espresso and more for test automation. You can also use AWS' existing test suite to test your application.
- Ability to test devices that are open and accessible to all or pay for a private test farm.
- Browser testing for Internet Explorer 9 and up, as well as the latest versions of Chrome, Firefox and Safari.
AWS Device Farm pricing starts at $0.17 per device minute for mobile testing, though users can start with 1,000 free minutes. Unlimited testing starts at $250 per month, and private device farm access starts at $200 per month.
Google Firebase Test Lab
Google Firebase Test Lab is another popular option for mobile testing in the cloud. Similar to AWS Device Farm, Google Firebase Test Lab offers testing with a variety of real mobile devices for iOS and Android. Its functionality is similar to testing on an actual device. Test Lab features include:
- Integration with the Firebase console, Android Studio and the gcloud command line tool.
- Compatibility with most continuous integration systems, such as Jenkins CI.
- Testing with real devices running within a Google data center. Each device is flashed and has up-to-date APIs, customizable locale settings and hardware configurations users can tweak to mimic real-world use.
- Executable Espresso and UI Automator 2.0 scripts as well as XCTests for iOS.
- Robo Tests for Android apps where Test Lab creates the tests for you. No need to script the tests, although you should review them to confirm they accurately capture your use patterns.
- Ability to distribute prerelease versions of the app to your QA testing team, which can make modifications or fix defects before you deploy a new version.
Google Firebase Test Lab offers a free version that includes 10 virtual device tests and five physical device tests a day. The paid version -- the Blaze Plan -- is pay as you go and charges $1 per device per hour for virtual device tests and $5 per device per hour for physical device tests. Google Firebase Test Lab only works with mobile devices and there's no option for testing with different browsers on nonmobile devices.
Why do you need so many test options?
One significant reason is your customer base. Do you know exactly what version and platform they use for mobile and the web? Do you know if they are still using Internet Explorer 9.0, for example? Whatever browser they use, your web application is expected to function flawlessly. The same is true for mobile devices. Many users stay with the phone they know and understand and expect mobile apps to function flawlessly regardless of device age or screen size.
Additionally, the number of dependencies on remote services and interconnected apps is steadily increasing, so an application needs to work on as many devices, platforms and version combinations as possible.
Microsoft Visual Studio App Center Test
Don't count out Microsoft Visual Studio, which offers App Center Test -- or Test Cloud -- for mobile testing in the cloud. This service features:
- Test automation services for both native and hybrid apps, similar to AWS Device Farm and Google Firebase Test Lab. App Center Test works with Appium, JUnit, Espresso, Calabash, Xamarin, UITest and XCUITest.
- Customizable device models and platform configurations to execute your existing automated tests via the App Center Test's command-line tool or the public REST API.
- Configuration settings to run tests each time code is committed against your customized list of devices and platform configurations.
- Test execution storage that can include screenshots and device error logs. Data is stored for six months.
- CodePush with which QA testers can get defect fixes directly to their test device.
If you already have Visual Studio, you can use the free tier, which includes 240 build minutes every month. There's a 30-day free trial available for unlimited testing on a single device. After the free trial, App Center Test costs $99 per month per test device concurrency. Similar to Google's Test Lab, Microsoft's App Center Test only supports mobile devices, with no browser testing offered.
AWS Device Farm vs. Google Firebase and Visual Studio App Center
In my experience, developers are impressed with AWS Device Farm, even though it's often too expensive for extensive QA testing. Many users find it easy to use and access. Google developers prefer Test Lab for the same reasons.
Microsoft's Visual Studio App Center Test is a complex, interwoven option with endless possibilities if you can arrange it optimally. Developers and testers who are already familiar with Visual Studio will find it easier to use than open source tool sets. However, unless all your applications are mobile, it's difficult to justify the cost of Test Lab and App Center Test due to the lack of browser support.
The best way to sort through these mobile testing tools is to create a test suite that you can execute within the free trial period for each tool. That way, you can compare the test execution, results, storage and automated tool integration across the board using the same exact test suite. Assemble a small set of testers unfamiliar with a tool and determine how easy it'll be to use for your team.
And of course, consider the cost. Determine the amount of testing you can afford. Use the tool for your most critical tests and, ultimately, decide which platform option would be the best bargain for you and your application's quality.
