QA Sanity Check: Multi-Device Testing For Mobile
Just a few years ago the Software Quality Assurance world was collectively rolling its eyes about the challenges of testing in an Agile environment. The current eye-roller within the QA community would have to be mobile device testing. Mobile is everywhere. If you, as a QA tester, haven’t yet been tasked with testing a mobile application, chances are you soon will be.
Unfortunately, QA folks are notoriously slow to adapt to changes in the technology industry. After all, we’re the gatekeepers, the voices of reason in the often chaotic development lifecycle. We are naturally hesitant to add new processes and methods to our tried and true testing toolbox. It’s the way we work. Our job is to be skeptical.
How do we even start thinking about testing for mobile devices?
Well, there’s iOS obviously, but what versions are relevant? Android is everywhere, but on what devices should we test? Does anyone use Blackberry these days? Windows Mobile? Samsung or Apple devices? Does WebOS even exist anymore?
Now that we’ve boggled your mind with the possibilities, let’s get practical. Here are a few quick tips to check your sanity when it comes to mobile testing.
NARROW YOUR SCOPE
Chances are the application or website you’re testing isn’t going to be used on every single device ever made.
If you’re testing an e-reader, focus on tablets. Evaluating a GPS most likely means you should center your attention on smartphones and handheld devices. Get some user data. Find out who’s using the application, or similar applications, on what devices. I once had a client tell me that they only wanted their app to work on five particular models. Talk about a narrow scope! Getting this information up front can make planning much more manageable.
Don’t forget that both iOS and Android have great PC-based emulators. While it’s certainly fun to go out on a mobile device shopping spree, it’s not always necessary if you can model the device in an emulator for basic testing.
RESEARCH OS AND BROWSER USAGE
There are many useful sites that chart OS and browser usage. If you don’t have your own user analytics, take a quick look at some of these sites. For example, gs.statcounter.com provides a wealth of information that can be used to make an educated decision on who uses what and where they do it. If .001 percent of the user base is using a particular configuration, chances are it shouldn’t be the focal point of your testing.
AUTOMATE IF IT MAKES SENSE
Test automation for mobile devices is tricky. For example, an automation tool that can test the perceived responsiveness of an iPad app does not exist. Usability and “feel” are incredibly important for mobile devices but test automation stinks when it comes to evaluating those aspects of the mobile experience.
If you want to ensure that your accounting app generates the monthly expenditure forecast correctly, or that your mathematics app correctly solves Fermat’s Last Theorem, there are plenty of new mobile-specific automation tools available.
One that I find particularly interesting is SeeTest by Experitest. They have created a record/playback automation tool that can also be used as a QTP or TestComplete plug-in. You can record and run your automated scripts on a PC, and then trigger them to run remotely over USB on your mobile device.
This is just one example of the many creative mobile testing solutions being released on a regular basis. Keep an eye out for any tool you can add to your QA arsenal.
THE PROCESS REMAINS THE SAME
Lastly, do not discard your existing QA process. Just because testing for mobile might involve some new considerations, it doesn’t render your existing process completely invalid. The best QA practice can be plugged into any website, application, device, hardware, or software, and should still output results with an equal level of quality.
Adjust your scope, adjust your approach, but keep in mind that you’re still tasked with finding bugs and handling discrepancies – just in a slightly different environment. QA testing in a mobile world is just a slight variation on the same familiar theme.
Michael Hollinger, Senior QA Specialist
With over 12 years of experience in the software development industry, Michael brings a wealth of experience having been a quality assurance analyst, lead, and manager, building and leading QA teams that utilize both manual and automated testing techniques. He has worked on various eCommerce, credit reporting and content management projects and clients, the most recent being Playboy Enterprises where he was the QA Lead for the redesign of the Playboy website and other signature websites, as well as their billing and subscription systems.