Application development for mobile phones: Testing across different devices

When writing an application for a mobile phone, do I have to worry about the differences in actual devices? How could I possibly test on all the different phones that are available?

Sorry to say it, but yes, you do need to be concerned with different devices. It is pretty well impossible to test all the devices, so it’s even more important to have a strategy to maximize the time you have. Here are a couple of factors to consider:

Operating systems: There are only a few operating systems on mobile devices so pick up test coverage by planning to test at least one device for each operating system. Symbian, Android, iOS (iPhones) and Blackberry are the current top three operating systems. But based on recent reading and research, I’ve learned the percentage of operating systems varies greatly by country, so look closely at where your users are located. The built-in browser varies with the operating system so this is something else to take into consideration.

Models: As I’m sure you’re aware, there are many models per carrier and every few months newer devices come to the market. Based on experience, the lower-end devices encounter more issues -- particularly in areas such as secure pages and user interface (display) issues. So if time is limited (and it nearly always is in testing), consider testing a low-end model from each manufacturer.

Manufacturers: Nokia, Samsung, HTC, Blackberry and the list continues. The market is large and continuously expanding.

For one client, I had access to Device Anywhere, which provides online access to a large number (depending on your paid-for package) of devices. For another client, we used a rather small assortment of devices available within the team (this reminded me of the days of performance testing where we did the best we could by coordinating people to test simultaneously -- an ineffective but cheap alternative).  A couple of people I’ve traded testing stories with have told me their project approach was to create a relationship with the key manufacturers they were interested in -- and then gain access to a set of devices. This is likely the most challenging aspect of testing with mobile devices.

Carriers: Personally, I have not encountered issues based on a device’s carrier. Here in the States, I’m familiar with carriers such as Verizon, AT&T and T-Mobile, while with European devices I’ve become more familiar with O2 and Orange. But it is conceivable that issues could vary based on carriers -- I would suspect SMS text messages might be more vulnerable to issues than web applications or websites -- but I do not have any data or experience to support that hunch (at this time.)

Statistics: I’ve given this recommendation in many areas of testing but it’s worth repeating -- look to any logs for statistics on your audience and the phones in use. If your application is just expanding to the mobile market, ask and help where you can to plan for tracking this data. Like other forms of testing, I strategize coverage based closely on usage.

Mobile testing, like other types of testing, can feel daunting and intimidating in the beginning. But break down what you need to do, arrive at a plan and do the best you can. Don’t let a daunting task keep you from starting.