Gartner says that more than 6.4 billion Internet of Things (IoT) devices will be in use in 2016, and that number will grow to more than 20 billion by 2026. Testing these devices—which range from refrigerators that automatically place orders to the supermarket, to self-driving cars—will be one of the biggest challenges to face device manufacturers and integrators in the coming years.
Effective testing is critical. But what's the best approach? Here are some considerations for testing IoT devices, and some tips that you can use to help you address them.
Tester's challenge: Too many IoT platforms
Each IoT device has its own hardware, and relies on software to drive it. Application software will also integrate with IoT devices, issuing commands to the device and analyzing data gathered by the device.
Because there are so many variants of software and hardware for devices, as well as different versions of firmware and operating systems, it might not be possible to test all possible combinations of hardware and software. To define a reasonable subset that can be tested effectively, gather information from your end users to understand which devices and software versions they’re using, and analyze it to determine the most popular combinations.
Once you know which devices and operating systems to test, you can focus most of your testing on these combinations, and run smaller sanity tests on less common combinations.
So many IoT communications protocols to test
IoT devices today use many different communications protocols to interact with controllers, and with each other. Protocols such as Message Queuing Telemetry Transport (MQTT), Extensible Messaging and Presence Protocol (XMPP) and Constrained Application Protocol (CoAP) are common, and each has its own advantages and disadvantages. But MQTT, which performs well in high latency and low bandwidth situations, is the most popular.
An API, typically based on JavaScript Object Notation (JSON) or XML, is often layered on top of the transport protocol to make it easier for developers to interact with the device. Testers can use the protocols and APIs for automated testing, and the test design depends on which APIs and protocols are being used. The testing tool must be able to support these protocols and APIs in order to be effective.
New attack surfaces and other IoT security threats on the rise
In October, a huge distributed denial-of-service (DDOS) attack on Dyn’s servers brought down giants such as PayPal, Twitter and Spotify (among many others). The attackers had infected thousands of vulnerable IoT devices with malware that caused the devices to generate traffic.
With more than 70% of IoT devices currently vulnerable to security issues, testing for security holes is a critical activity. IoT device testers should pay particular attention to the device’s password policy, making sure that minimum password requirements are built into the device, and that they are enforced.
A recommended practice for devices is to require a password change upon first access, and this should be taken into consideration when developing automated tests.
IoT app, device diversity a challenge
The fact that the types of IoT devices and applications are so diverse demands strong test capabilities. Performance must be consistently high across all devices, and exceed user expectations.
Testers must have a strong test strategy, have a good understanding of the architecture, and ensure that the devices and software under test are always configured with the correct version. If the system depends on third-party services, tests may fail if that third-party service changes. Automated tests that run as part of a continuous testing pipeline will detect this very quickly.
If the third-party service is unavailable, you can virtualize the service, and remove the dependency on the actual service. If the service is down, you can still test by using the virtualized service. A customizable service virtualization tool will let you configure the expected responses from the service, so that you can test your application’s response to various situations.
Fast-moving data and increased load take a toll
Connected IoT devices rely on fast communication. Consequently, network status can have a significant effect on device performance. Smart devices often experience problems with network infrastructure, such as overburdened WiFi channels, unreliable network hardware, and slow or inconsistent Internet connections. IoT devices and applications must be tested across these different conditions to ensure that they respond correctly without losing data.
Creating a network with all of the different conditions is rarely practical, so many teams use network virtualization to programmatically emulate different network conditions and responses. Device system metrics, such as CPU and memory, should be monitored during testing.
For example, an IoT device that has a memory leak will perform slowly, and may not respond correctly. Software on desktop or laptop computers is activated and tested by the user tapping, clicking or typing. But IoT devices are often passive, so testers must understand what devices are being used, and know how they behave. This requires an adjusted look at performance testing tools and performance monitoring.
With IoT complexity comes opportunity
While the IoT brings a new level of complexity to performance monitoring and testing, the business opportunities that the devices present nevertheless offer an exciting new frontier.
Testers must adapt to new platforms and techniques to ensure that they are able to address the challenges of testing IoT devices and applications in order to deliver the best experience to the end user. Fortunately, improvements to network infrastructure and testing tools will continue to make the process easier over time.
Image source: Flickr
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
