You are here

Mobile app testing: When to use real devices versus emulators

public://pictures/Will-Kelly-Freelance-Technology-Writer.jpg
Will Kelly, Freelance Technology Writer, Will Kelly Writes

Should you do mobile app testing on real mobile devices, use emulators, or do both? The answer depends on what you want to test. For example, emulators are good for user interface testing and initial quality assurance, but real devices are essential for performance testing, while device cloud testing is a good way to scale up the number of devices and operating systems.

Here are some pros and cons to think about when considering whether to choose real devices or go with emulator-based quality assurance (QA) testing.

World Quality Report 2018: The State of QA and Testing

Real versus emulator: The trade-offs

Lubos Parobek, vice president of product at cloud-based testing platform provider Sauce Labs, says emulators are a great place to start because they're less expensive than mobile devices. "They're faster, and they catch the most common bugs," he says.

"Emulators allow you to debug flows as you're testing the app," says Ahmed Hassan, senior QA and test automation engineer for collaboration platform provider Huddle. You can delve into code, file structures, and databases in real time, allowing for nuanced inspection of whatever feature you're testing.

But Eran Kinsbruner, mobile technical evangelist at Perfecto Mobile, a mobile app testing platform provider, stresses that testing mobile apps on emulators only is a huge risk because your users are running apps on real devices, not emulators. There are things that you just can't find on emulators and simulators, and that's one of the biggest reasons that people use them in conjunction with real devices.

Real devices are of course good for testing performance, Parobek says. "How much of the CPU are you going to use for your app? How much memory? How responsive is it?"

And it's not just device performance you need to worry about. "Real devices are good for testing your application under less than optimal network conditions," he says. "How does your app respond if the network is dropping in and out, for instance?"

Physical inputs are another big challenge. "With all these sensors such as GPS, proximity sensors, life sensors, force touch gesture, make sure that your application responds properly," Kinsbruner says. "Many of these gestures cannot be simulated. If they can be emulated, it's probably going to be only on one flavor or one emulator, not on all the LG, Samsung, and Sony emulators, and so forth."

Most importantly, emulators may not be able to test your app at all. "There are a bunch of third-party libraries, for example, that require a specific type of hardware, and that can't be run on an emulator," says Andrej Kostresevic, CEO at the mobile app development company Nomads.

[ Webinar: World Quality Report 2019: Focus on the Financial Services Sector ]

What device cloud testing delivers

Setting up a mobile environment that is always on, always up-to-date with new devices and operating systems, and available for testing 24/7 is a difficult process. A device cloud can be hosted on-premise, but more typically it will be hosted in the public cloud, where you can rent out devices for testing your mobile apps on a one-time or subscription basis. Some emulators are also making the move to the cloud.

"If you want to just focus on quality and releasing mobile apps to the market, and not deal with device management, let the cloud do it for you," Kinsbruner says. Amazon, Google, and other cloud service providers have offerings for customers who lack the resources and skills in house to manage and secure mobile devices in a cost-effective manner.

These providers often have access to the latest mobile devices and can offer them to customers who want their mobile apps to be tested against the most current enterprise mobile devices.

Moving to the cloud lets you focus your staffing and resources on your mobile application road map. But, Kostresevic warns, "Even with a device cloud, it's not feasible to test on every single device." He advises tracking device popularity, noting that 10 percent of devices out there represent 90 percent of the market.

While you can do this work in house, it may be easier just to go with a vendor that specializes in cloud testing and already has everything set up. "At any point in time, there are probably 20 to 50 devices that give you really good, representative coverage of things like screen sizes, processing power, and memory," says Kostresevic.

Android developers can get the diversity and depth of Android operating system versions to support mobile app testing from a device cloud provider that represents the target user base for the app.

And the winner is...

As you've guessed by now, choosing between emulators and real devices for testing shouldn't be an either/or proposition. Your decisions should center on finding the right balance of emulator and device-based testing to support your QA standards throughout continuous integration and deployment.

Huddle's Hassan says he makes the emulator versus real device testing decision on a case-by-case basis. For example, changes that impact both the user interface and mobile app architecture are best tested with a combination of physical devices and emulators.

The emulator versus real device decision also depends on where you are in the development lifecycle. For example, you might use an emulator to verify the look and feel of a mobile app, then move on and do other tests on real devices. Device clouds, he says, are a solution for large-scale device testing. Start with two to three devices on premise, he advises, and then scale up as your operation grows.

Your takeaways

Here's your executive summary: use an emulator to test the user interface of your mobile app and to debug flows as you're testing, but remember that gestures can't be tested on an emulator. For those tests you'll need to use a real device, as well as for testing app performance under less than ideal network conditions. Scale up your real device testing by outsourcing to a device cloud provider, so the provider can manage device procurement, setup, configuration, and security. Ultimately, you should decide whether to run mobile app tests on an emulator or a real device on a case-by-case basis, factoring in elements unique to your organization, such as budget, staffing, project time lines, and a host of other factors.

Here are recommendations for choosing between an emulator, real devices on-premise, or a device cloud:

  • Use an emulator
  • Use real devices
  • Scale up your real device testing by outsourcing to a device cloud provider where the provider manages device procurement, setup, configuration, and security.
  • Decide upon using an emulator versus real devices for mobile app testing on a case-by-case basis, or seek a logical step in your development cycle where emulator testing.

How do you combine emulator and real device testing in your mobile app testing efforts? Add your comments below.

Image credit: Flickr