How to take a systems approach to mobile app testing

One of my worst testing failures occurred when I worked at a healthcare company that offered a mobile heart-monitoring app. While my team had thoroughly tested the GUI, we didn't account for interactions with the overall system environment. Then we discovered that the app overheated the medical device. That caused the cellular modem to fail, so patient data wasn't sent. It could have been catastrophic.

With mobile app testing, people focus on the functionality of the software and GUI, rather than the entire system. That’s a big mistake. You need to use systems thinking to incorporate broader test coverage while exposing areas in need of a stronger design. But how do you apply this approach and develop tests specifically for mobile application development?

Here's what you need to think about while creating your test cases and strategy, along with a few real-world examples of what can happen.

Testing in the Agile Era: Top Tools and Processes

Start with your test strategy

Does your mobile app test strategy incorporate hardware and operating system firmware conditions to better understand software behavior, or is it focused on software only?

Mobile apps rely on the hardware to perform functions such as notifying the users of information. And sometimes the firmware-based operating system influences software functions, such as how memory is used.

So what tests should be included for mobile app development? And how does a team start to develop system-level tests to achieve more thorough test coverage? Testers are skilled at writing automated tests, but those tests do not always incorporate systems-level thinking. Software testers and managers tend to focus on software functionality, along with the app's look and feel. They're not trained to consider the entire system. Often, mobile tests are seen as quick and easy to automate. They're not. 

Don't fight the system

Systems thinking must start when mobile app test engineers first start to formulate their test development strategy. You need to know what the app will be used for, the types of users who will use it, and the skills those users need to use the mobile device. Those details can be gained though your app's analytics.

Knowing as much about your users as possible helps testers create more appropriate tests. You want to know how people use your mobile app and when they expect to see specific functionality.

One approach I find helpful is to ask yourself the who, where, how, and when with regard to functionality. For example, what happens when you are using an app that's image-heavy, such as a GPS/navigation app, and you need to also charge the device? Does the device get hot? Is the device in a protective case? Just using the device itself generates heat from battery usage. Running apps add to that.

Charging the device while it's in use generates even more heat. That's important because the battery sits very close to the cell modem inside the mobile device. If the battery gets too hot, it can damage the cell modem. So your app should build in a periodic check of battery temperature. My suggestion is not to exceed 80 degrees Celsius. If the device exceeds the specified temperature, you can configure your app to shut down the cell modem until the battery reaches a cooler, less dangerous temperature. 

As a tester you can not only inform the development team of the conditions but also assist in creating a design that prevents the system from producing an unacceptable result. Testers should consider the conditions of usage and spend time exploring the implications of that for a given user profile. And when I say to understand the conditions of the device I also mean how the app itself uses the device’s hardware and firmware.

Know your hardware limitations

Understanding the hardware's limitations can help developers design stronger apps and avoid pitfalls. Mobile app test engineers can participate in that process by providing more information about the app's behavior under stressful conditions.

Learning the boundaries of the hardware conditions can help your development team prevent crashes or stoppages where no error message indicates what is going wrong. How does your app behave when the cache isn’t quite cleared out but the app is reporting some data that isn't what you'd expect to see after a restart?

At work recently, a system integration tester ran into problems with an app that lets passengers on an airplane access the Internet and stream TV shows and movies. The test engineer found that a mobile device couldn't connect to the in-cabin network.

The plane connects to a satellite by way of an antenna that in turn connects to an onboard router. The app must establish a connection between a passenger's mobile device and the router. It’s a fairly complex system, so testers need to understand how the entire system works, including the firmware on the router and any interdependencies. The key to learning more about the problem was discovering not only that passengers couldn't connect, but also when that was happening and where each passenger sat on the plane when it occurred.

The problem was intermittent, which added another level of complexity. The integration tester did some digging, only to find that the onboard router's firmware was old and included a patch that wouldn't assign a unique ID. After upgrading the router to a new firmware version that corrected the problem, the tester had to come up with tests that ensured not only that the firmware change didn't affect the rest of the app’s behavior, but also that the device could connect to the plane’s in-cabin network with a unique address.

As a tester, you need to think about performance, too. But performance is not just about how fast you can make the connection. Especially with mobile devices connected to a contained network such as an in-cabin router, you need to think about how many mobile devices can connect at a time. Is there any effect on app usage as more people try to connect? Are all connected device ID addresses unique? How many devices can connect at any given time? Can the firmware handle an aircraft full of passengers accessing the network? Does the firmware know to switch routers when it reaches capacity? Knowing the limitations of the hardware, as well as the decisions the firmware makes based on the limitations of the hardware, is important for mobile test development.

Your mobile app may not be connected to such a complicated system as an aircraft, but every mobile smartphone is in itself a system, and a complex one at that. So as a mobile tester, you must think at a system level. You need to understand the type of mobile architecture on which the app your test was developed.

Does the app take advantage of the device’s hardware, such as using a flashing LED light or audio for notifications? Which LED light color should be used for each type of notification? Does your app allow use of the camera? What about voice commands? Each of these things needs to be tested under many different conditions, both for new apps and for any changes to your existing ones.

Think about the what and the when

The key here is to constantly ask what is supposed to happen, and when. That, in turn, depends on how the app is used, along with any interdependencies. The "when" question isn’t about the speed of a given function, but whether it makes sense for that function to occur, or to be allowed while the device is in a specified state. Ask yourself the question, “Does the functionality make sense, and will it work based on the conditions of the system?” This is never easy to derive.

As a mobile tester, you are no longer just a software tester. You need to be a systems thinker and systems tester. To give strong, high-quality test coverage, you need a systems perspective and knowledge as to how your customers will use that system.

What to know more about taking a systems approach to mobile testing? Join me at the online Testing Guild Conference on June 20-22, where I'll conduct a session on building a mind map with the audience to help inspire systems thinking for mobile test development.

Testing in the Agile Era: Top Tools and Processes
Topics: Dev & Test