For development teams just entering the wonderful world of mobile application development, mobile application testing can be full of unknowns. Developers and quality engineers may find themselves wondering what they've forgotten to test against, or worse -- missing bugs because they didn't know to perform certain tests. The App Quality Alliance is pulling together data from its members to create testing guidelines that will give developers confidence that they are testing mobile apps thoroughly.
The App Quality Alliance (AQuA) is a non-profit group working with the industry to improve the quality of mobile apps. Its members include AT&T, LGE, Microsoft, Motorola, Oracle, Samsung and Sony Mobile. SearchSoftwareQuality.com caught up with Martin Wrigley, executive director of AQuA, for an interview at the Mobile App Europe conference in Potsdam, Germany.
SearchSoftwareQuality.com: In your presentation, you spoke of 10 poor practices still in use by many app developers. Can you describe a couple of the most insidious?
Martin Wrigley: Apps need to work well in a real environment, in customers' hands. But many developers use ideal environments for the functional testing of their [apps]. This may mean that they have the fastest, newest phone, or that they only use Wi-Fi connectivity, or that they forget that real users fill up their phones' device memory with other apps, photos, music and messages.
Developers don't always remember that a device can lose connectivity -- and their app doesn't necessarily behave in a way that makes sense to the user when that happens. It isn't good for an app to freeze when the device is set on airplane mode -- but that still happens all too frequently.
There is no substitute for having a separate team (apart from the development team) test an app on real devices in a real situation before giving the app to users.
Tell us about AQuA's app testing criteria. How were these developed?
The testing criteria are reviewed on an ongoing basis, and new versions are released usually two or three times a year, as necessary.
Wrigley: AQuA has criteria for Java, Android and Apple iOS apps. We are working on Microsoft criteria as well. The criteria are designed to catch the majority of common errors in apps. These criteria have been developed over time, mostly in response to errors that our members have found in apps. As we find new trends, such as one recently where apps were neglecting to release the camera on devices, we add to or amend the criteria to catch these trends.
Our member representatives are involved in apps on a day-to-day basis, either testing apps on devices, looking at the impact on the network, or delivering apps to paying customers and having a reputation for quality.
The testing criteria are reviewed on an ongoing basis, and new versions are released usually two or three times a year, as necessary. We receive a good amount of feedback from the thousands of developers who have signed up to receive information and updates from us, and we ask for new versions of the criteria to be reviewed by them before we make updates public, in much the same way that you should have someone test your app externally.
AQuA members debate each new addition to the testing criteria. We are keen to ensure that they remain "right-sized" and relevant for apps that are downloaded today. We consider the criteria to be a complete but minimum set of tests needed to ensure a good customer experience.
How will following these criteria improve app quality?
Wrigley:If a developer follows the test criteria, he will avoid the top 10 errors and more. The testing criteria give developers a QA system in a box. Since the launch of the Apple App Store, we've seen the introduction of tools that could be used by designers as well as developers. We are seeing apps that are much more attractive to look at, but the teams behind them may not necessarily have the software engineering skills that the traditional development teams may have had. As a consequence, they may not have the testing expertise to look beyond the immediate functional testing that people tend to do.
The AQuA testing criteria gives developers a cost-effective way of performing a structured set of tests -- a checklist -- to avoid easily overlooked errors that otherwise keep causing issues in the apps.
What testing advice do you have for teams building their first mobile apps?
Wrigley: In the planning of your app, think about the types of tests you need to do and don't skimp on them. It's always tempting to put in another feature and miss out on some of the testing, but if that ends up with your app crashing, that extra feature will not be much compensation.
