Walk through any department store, home improvement center, or electronics store—or browse Amazon.com—and you will see an unending variety of connected smart devices. Gartner forecasts that 20.4 billion Internet of Things (IoT) endpoints will be in use worldwide by 2020. In 2017 alone, 7.2 million new devices will get connected every day. For many IoT apps, performance is critical. But performance testing of IoT apps is a bit different from that of traditional mobile and web apps.
IoT devices serve as a bridge between endpoint devices, mobile apps, and the cloud that makes it all work. Building and maintaining those apps has led to the DevOps and microservices architecture world we now live in, with organizations using continuous delivery (CD) and continuous integration (CI) to deliver and update applications to demanding consumers.
For many IoT applications, performance is one of the main risks in the development project, especially in areas such as healthcare, where personal safety could be affected. But for traditional performance engineers used to testing pure web and mobile applications, the IoT presents some new challenges.
Go to TB LearnMobile APM: An introduction to mobile performance monitoring and testing strategies
What makes IoT architecture different
Before getting into the specific performance-testing challenges, it's important to review a few fundamentals of IoT architecture. (If you don't need a refresher, skip ahead to the next section).
The IOT architecture
An IoT architecture has three primary layers: the endpoints, the IoT platform, and the apps.
The endpoints and IoT gateway
Endpoints are the physical hardware sensors. Each sends messages to the IoT platform through a gateway. The gateway is essential, because objects could potentially move and lose connectivity (such as when sensors are moving, on a ship, train, bus, bicycle, or runner). The role of the gateway is to store messages and deliver them to the IoT platform when the device reconnects. Each sensor connects to the gateway through one of several data link protocols, each of which offers a different mix of short- and long-range and low- and high-bandwidth options.
[ Special coverage: PerfGuild performance testing conference ]
The IoT platform
The IoT gateway uses session protocols to communicate with the IoT platform, the common layer from which the enterprise handles many applications and thousands of sensors. The main role of the platform is to store and analyze the data the sensors sent to it. It is, in a sense, just a big data architecture.
The IoT platform stores and processes the information for consumption by the customer-facing IoT application.
The application layer
This represents the last mile, where several types of users might consume IoT data. The application layer usually has several end-user apps, such as a public web or mobile app, and back-office applications, such as CRM.
The application layer uses the HTTP transport layer to exchange data between the IoT platform and those apps.
IoT app performance-testing challenges
One of the main challenges in testing IoT apps is the technical environment, where performance engineers must decide on a design scenario. Specifically, where do you find a load-testing product that supports the latest protocol used by your IoT app?
Use a load-testing tool that includes an SDK, so you don’t have to rely on your software vendor's R&D organization to develop what you need to load-test your app. You will be able to code your own protocol and use it within your own project timeline.
For protocols that use the low-level transport layer (UDP or TCP/IP), you want to be able to take advantage of the recording engine of your load-testing tool. But forget about the record/playback approach. Each gateway will subscribe to different topics and send a different type of payload. Correlation, and designing the exact workflow, will be required. If you capture the traffic and just replay it, you will probably generate errors, and the message generated won't reflect the real situation.
Gateways and sensors have a precise workflow of messages. They send data every x seconds and can receive orders that cause the gateway or sensor to send additional messages. You must fully understand that workflow before designing the behavior of the endpoint.
As with traditional load testing, it’s important to understand how users and objects interact to create meaningful scenarios for the entire IoT architecture, from the application layer all the way down to the endpoints.
Network constraints are an important piece of your emulation, because gateways and end users are connected to a restricted network, and that introduces latency and packet loss. Testing without the network will hide potential issues on the sizing and configuration of the environment. So don’t forget to use wide-area network emulation tools or load-testing tools that include this feature.
Where to focus your load-testing efforts
Depending on your IoT architecture, performance could be one of the main risks of the project. That's why your testing strategy should be based on a risk assessment.
Going back to the IoT architecture diagram, you can see the areas where you should focus your testing efforts. These include:
Sizing of the IoT platform
Because you usually have a common layer for all your objects and IoT applications, you must validate the sizing, scalability, and overall performance of the platform. In order to update your architecture, you need to identify the current limitations of your IoT platform.
Test engineers often think performance is not an issue for the IoT platform when it's hosted in the cloud. Outsourcing the hosting environment on a cloud platform doesn't mean you won’t face performance issues. Even if you have an elastic architecture, you must fine-tune the thresholds that tell the platform when to spin up new nodes (or containers) within your architecture.
The IoT platform is the heart of your IoT architecture, so test it as early as possible. Test the main components (or API) of the IoT platform on a daily basis to identify potential regressions, and then monitor the performance levels between each build.
Sizing the gateways
The gateway handles communication with all of the local sensors. So, depending on your implementation, you could have a gateway that handles a few objects or several hundred. Gateways are usually a Raspberry Pi or other small mobile device with limited hardware. That’s why you need to identify the limits and the capacity of each device. The gateway is the main node that you use to interact with your objects, so don’t underestimate this point.
The testing of gateways is the most complex piece of the performance puzzle, because you need to have a load-testing tool with a load generator you can use to send messages over data-session protocols using Bluetooth or Wi-Fi.
The solution is to:
- Build a Raspberry Pi device that supports the expected protocol.
- Select a load-testing tool whose load-generator component can be installed on a small Raspberry Pi device.
- Code the expected protocol using the SDK associated with your load-testing tool.
You also need to define the response time for your IoT app. But here, simply measuring the time to send a message to a broker doesn't make sense. It's better to measure the time required to travel across the entire architecture—that is, the time required for a message to travel from an object to the end user, or vice versa. There isn't a standard solution for measuring this key performance indicator.
Also, you should be able to change the payload of a message by adding a departure timestamp. With the help of this timestamp, you can calculate the response once you receive the message.
Other performance-testing challenges with IoT
An IoT architecture comprises a big data platform and several app end users connected to the IoT platform.
To generate a sufficient real-world load against that platform, you need to have representative data stored within it. Don’t underestimate this variable, since it will change the results that you get during your tests.
But having the full environment, including endpoints, the IoT platform, and your applications, available for testing won’t be possible. So be smart:
- Simulate the load against the IoT platform by simulating the traffic of the gateways on the applications using the main APIs or components of the architecture.
- Measure the user experience (UX), which means simulating the noise against the IoT platforms (gateway application traffic on the main APIs and components used by the application to consume data). And you need to take those UX measures from a browser-testing or mobile-testing tool.
5 steps to success
To sum up, performance testing an IoT architecture presents challenges for traditional performance engineers because of the complexity of the environment. Here are the key points to take with you:
- The protocols typically used aren't supported by most load-testing products, so don’t wait for the tool vendor's R&D team to support your IoT stack. Select a tool that includes an SDK so you can start building tests ASAP.
- Your testing strategy should be based on an assessment of your project's risks.
- Your IoT platform is the heart of your IoT architecture, so test it as early as possible, and monitor the level of performance between builds.
- Having a complete testing environment for IoT performance testing isn't possible because there are simply too many dependencies. Mocking and simulating end users' applications and the traffic on the gateways will help you generate a representative load against your IoT platform
- To test the performance of your end users' applications, you must simulate the load against the application and your IoT platform. And on the other end, you need to simulate end-user application activity by consuming the APIs and components available on the IoT platform.
Performance engineers who follow these five tips should have no problem adapting their testing practices to their IoT architectures. These are my best practices. What are yours? Testing IoT platforms will be common practice in a few years. I am very interested to hear your best practices as I continue to build my own IoT performance-testing checklist. So don't be shy; post your comments, and let's build this initiative.
For more on performance testing and IoT, participate in my live online presentation at the Performance Guild virtual conference. And if you miss the live presentation, you can still watch it after the event.
Image credit: Flickr
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
