Checklist for mobile app testing: 15 gaps to look for

1. Finger-clickable links

On a mobile device, a directory listing by last name can be frustrating -- if not impossible -- to accurately click with a finger, even if it works easily on an emulator or by an automation tool. For instance, the capital I and lowercase i can both render as a single pixel in width.

Apple Music, for example, uses bold letters that are two pixels wide to avoid this problem. Test how easy it is to scroll through and select individual letters within the application's directories.

2. Dropped wireless service

Imagine a mobile app user who is about to -- or just has -- hit the checkout button on a transaction, but then loses cell service. Consider how frustrating it would be for the user to not know whether the app processed a transaction. Test how the application retains and recalls transaction information when a user regains service. If the application has the functionality to notify users that their action failed, check that feature as well. You can save customers the frustration of accidentally making the same purchase twice or from exasperation that leads them give up entirely.

For example, YouTube and a few other applications turned this problem into a feature. These applications allow users to cache videos and play them offline -- for a monthly subscription fee.

3. External interruption

Test applications to see how they function when interrupted by a call, text message, alarm or third-party notification, such as from Google Maps. One method that works for most teams: Try a phone call long enough to force the software to time out.

4. User-based interruption

Many mobile applications go into passive mode when users put their phone to sleep or navigate through other apps. Some applications let users play podcasts or music in the background while immersed in something else; other apps do not allow this. Placebo by Momanda, for example, can't play music if the user's phone goes to sleep. Also, the YouTube iOS application will play only if it's full-sized. This means users will face interruptions if they have a low timeout setting or if they put the phone to sleep as a habit. Verify the developers' chosen approach to app function while not in primary-use mode to see if it works correctly.

This is another case where YouTube turned a problem into a feature. It offers a paid service to play audio when YouTube is in passive mode or the screen is locked.

5. Battery drain

Testing battery drain doesn't have to be expensive; it can almost be free. Simply test the application all day on Friday -- i.e., leave it on all day -- and observe battery drain. Use the same phone Saturday and compare battery at the same time as you did on Friday. If there's a significant difference, you may have a problem.

Consider Highlight, a location-sensitive mobile app that connects users if they have something in common and are nearby. Its core functionality requires a constant back and forth with servers, which leads to extensive battery drain. Highlight was once praised at the SXSW Interactive Festival but it's no longer a serious player in the space.

6. Cellular data drain

Cellular drain is like battery drain, only scarier. Firstly, cellular use only shows up after it's drained. Lucky users get a text when they use 75% for the month; unlucky ones get an unexpected bill. One expensive bill -- or sudden loss of wireless service mid-month -- could be all it takes for users to quit your application.

This is the kind of problem that a simulator will never find, and neither will testers who test only on their business or personal Wi-Fi networks. However, you can find this just as easily as battery drain -- check bandwidth, test for a day, compare that to the bandwidth the app used on a Saturday and extrapolate demand. The 10 minutes it takes to record these numbers could be important.

7. Excess heat

Processing-intensive applications -- such as video players and non-native software -- can generate a lot of heat. This reality is especially true for non-traditional uses, such as listening to videos as if they were audio-only on a long car trip. Another case might be habitually charging a phone while videos play or entertaining children on a long trip.

Excessive heat suggests an application uses a great deal of CPU, memory and disk; it's also likely draining the battery and cellular data. The app programmers may have made a functional mistake on a loop structure, which could lead to slow performance and memory loss.

8. Memory loss

Run the application concurrently with a variety of other apps on an older, low-memory mobile device. Look for any performance slowdowns or crashes. They will happen; the challenge is to then decide how much performance degradation is acceptable and what the oldest devices are that an app can reasonably run on.

9. Cellular speed problems

Another factor for data-intensive applications is response speed. What might work exceptionally well on a company's internal network could fail on a 3G network on the edge of cell tower range. Both Apple and Android ecosystems have utilities like network link conditioners that can simulate a slow, delayed or lossy connection during mobile app testing.

10. Cellular quality problems

The network link conditions can also simulate packet loss. This failure of data to reach its destination tends to happen at the edge of cellular networks or when too many users are on a system. Simulate escalations in packet loss as users try key parts of the system, including checkout or purchase. With 100% packet loss, the system will fail, but the question will be how gracefully it fails and how useful it will be to the user. It may be possible to restore the state of the transaction when the network "fixes" itself.

11. Deep linking

Take a link to a regular webpage, accessible only after login from a desktop, and email it to a phone. When the user clicks the link, does the app direct them to log in? If yes, test if users are sent to the specific linked-to page. And if it's relevant, ensure the app redirects users to the mobile version of that page. This practice is called mobile deep linking.

12. Portrait mode repaint

Some applications, such as Apple's Calculator, display more functionality when the screen turns. These applications repaint the screen, which causes a change to the form factor. See if a user can access these capabilities in practice.

13. Swipes and other motions

Don't overlook how fundamentally different it is for a user to interact with an application via their hands as opposed to a keyboard and mouse.

If you run an application with a testing tool, or even an emulator, it can feel much different than on the physical phone. Test the way someone would actually use the application. Problems that directly affect the user experience might be hiding in plain sight. For example, ferret out any app functionality that requires a right click and send it back to developers for a rewrite.

14. Distance

Users on other continents will expect your software to be fast. While you can simulate this with a VPN, it's even better to hire a freelancer or two on other continents. Crowdsourcing vendors makes this kind of testing cheap and effective.

15. Older devices

Test on the oldest phones and tablets that wireless carriers still support. Expect to see all of the problems listed in the above checklist for mobile app testing manifest earlier in the process. Older devices get hot. They crash and run out of battery faster than those manufactured today. For additional coverage, consider crowdsourced testing or a cross-browser compatibility tool. A few of these offer free plans. Most offer a free trial.

Put it all together

Software developers can spend days on this checklist for mobile app testing. As code releases happen more frequently, you may be tempted to test only a few of these potential failures or perhaps forget about this sort of extra-functional checklist for mobile app testing altogether.

Don't do that.

Instead, keep a list of risks that you periodically review. Changes in the browser or cellular market may make some of these bubble up and others decrease. If you'd like, make a deck of cards with these risks, shuffle them and pull out five to check with every deploy. No matter what you do, stay vigilant.