You are here

How to choose the right test framework for Android

public://pictures/biopic.jpeg
Satyajit Malugu, Senior SDET, SoFi

The Android platform, now more than 10 years old, has plenty of testing tools available. Java is even more mature, with a rich assortment of testing tools for Android projects.

Then you have open-source UI testing frameworks, including Appium, Espresso, Robolectric, and Mockito/Powermock, from which to choose. All of this can be confusing, especially if you're just getting started with Android.

Whether you're new to Android and trying to sort through what's available or you just want to get up to speed on your options, this overview of four key open-source testing frameworks should help you decide which best fits your testing strategy. 

[ Get up to speed on quality-driven development with TechBeacon's new guide. Plus: Download the World Quality Report 2019-20 for lessons from leading organizations. ]

JUnit is the foundation

First, let's start with the basics: JUnit is the default test framework for Android. Mockito, Robolectric, and Espresso all use it as the backbone and build on top of it. This is one decision you don't have to make. 

More specifically, AndroidJUnit4 is the current runner that provides utilities to run Android tests. After recent releases, AndroidJUnitRunner was deprecated, and AndroidJUnit4 became the single runner for both Espresso and Robolectric tests. 

As you might guess, there's an AndroidJUnit5 on the horizon, and you can get native support for JUnit5 style tests within Android. 

Appium

No discussion of mobile testing frameworks is complete without Appium, the native mobile testing framework that works with Android, iOS, the Windows desktop, and Internet of Things devices. 

When and why to use it

  • For very high-level, flow-type tests.
  • For testing on exact PROD apps as a user would. It provides the highest fidelity to your end-user scenarios. 
  • For integration with other apps, such as settings and dialers, and for system-level options, such as push notifications and widgets.
  • For teams that have similar apps across platforms and want to have a single test repository that can target all of them.
  • For teams that have web Selenium experts and want to leverage that expertise for native app automation. 
  • For teams that have hybrid or React Native app and Appium tests that can be co-located. 

Why it might not be a good fit

  • It's slow. Its round trip HTTP-based architecture means each call to find an element is slow. Native apps use faster locators.
  • Appium code typically is not co-located with mobile app repos and can quickly get out of sync. 
  • The framework is highly dependent on open-source contributions. As such, it might lack documentation and support for the latest releases. 

Recommended CI setup

  • Appium runs daily on a very small set of end-to-end high-value flows on multiple devices. 

[ Get up to speed with TechBeacon's Guide to Software Test Automation. Plus: Get the Buyer’s Guide for Selecting Software Test Automation Tools ]

Espresso

Espresso is the default UI automation framework for Android—even Appium uses it under the covers. While you had few alternatives in the past, Google has been steadily improving and investing in Espresso, making it unparalleled in support and adoption in the Android community.

When and why to use it

  • It's very fast because of its insight into an app's status.
  • It's a good choice for dev-focused teams that do not have any prior test automation experience.
  • It's co-located with dev code, which makes it easy for tests to be in sync with changes.
  • You can reuse helpers and mocks that you created for unit tests.
  • Google has a vested interest in it, and teams are working on making this framework forward- and backward-compatible across devices. 

Why it might not be a good fit

  • Your team is experienced with Selenium and is also responsible for UI automation on other non-Android platforms.
  • Your developers do not consider UI tests to be part of their responsibilities.
  • Your team doesn't want to maintain separate sets of tests for iOS and Android. 

Recommended CI setup

  • Create a small set of smoke tests that you run on every pull request.

  • Build a bigger set of regression tests that you run on every release candidate.
  • Use emulators on cloud labs.

Roboelectric

The Roboelectric UI testing framework, which went mainstream in the last few years, has parallels with Headless Chromium. Roboelectric has no physical UI, and tests operate in a simulated environment. Android SDK framework APIs are available, and view interactions are mocked. The biggest differentiator between Roboelectric and Espresso is that Roboelectric has no need for an emulator or device.

When and why to use it

  • You want to test your UI views without the overhead of emulators or cloud labs.
  • You want tests that are very fast (~2 sec.) versus merely fast (~10 sec.), as with Espresso.
  • You want tests that exercise the UI in various error and edge cases. 
  • You want to test activities and fragments in isolation.

Why it might not be a good fit

  • Your tests navigate from one screen to the other.

  • The setup required to mock a test is becoming heavy and cumbersome.

  • Your CI setup isn't scaling and is running into out-of-memory exceptions, with hundreds of Roboelectric tests.

Recommended CI setup

  • About 300 tests that run after pure unit tests.

Google is contributing heavily to Roboelectric, and with Project Nitrogen, the lines between Espresso and Roboelectric are blurring

Mockito and Powermock

Mockito is a mocking framework for unit tests written in Java. It's similar to Powermock, but provides additional mocking functions, such as singletons. Powermock's GitHub page describes it as "a Java framework that allows you to unit test code normally regarded as untestable." 

Unit testing mocking frameworks are ideally suited to view model testing and helper classes testing. In well-architected code, say following an MVVM or MVP pattern, about 50% of the code should be tested using unit testing frameworks. And these frameworks shouldn't need Android SDK hooks. 

When and why should you use it

  • Your code is modularized and componentized and can easily be mocked.
  • You want tests that are very fast (milliseconds).
  • Your team cares about code coverage metrics.

Why it might not be a good fit

  • You need test UI rendering.
  • Your tests need access to Android SDK functionalities such as lifecycle, intents, threads, etc. 
  • The setup required to mock a test is becoming heavy and cumbersome.
  • Tests become brittle, and each app change requires modifying test code as well. 

Recommended CI setup

For every pull request, run all the unit tests to provide instant feedback. Unit tests should cover about 50% of the code and are usually in hundreds. 

Putting it all together

You can visualize all these frameworks in a class test pyramid that combines the idea of how many tests are recommended at each level and their foundational nature. 

Source credit: Satyajit Malugu

Note that there can be exceptions to this general categorization. For example, you can use Espresso as a complete black-box tool if you don't use some of constructs provided in the framework. 

Now, get started

Google and the open-source community have done a phenomenal job of creating various testing frameworks for different levels of Android. With the information above as your guide, you can start customizing your own Android automation strategy. 

[ Learn how to apply DevOps principles to succeed with your SAP modernization in TechBeacon's new guide. Plus: Get the SAP HANA migration white paper. ]