Measuring the throughput of the IoT gateways

The key part of the process of tailoring the perfect IoT solution is choosing the proper hardware for the gateway device. In general the more expensive gateway hardware is, the more messages per second you can process. However the more expensive the gateway device is, the less affordable your IoT solution becomes for the end client. That is the main reason why would you like to have a proper tool for measuring the given IoT messages flow scenario in the unified way, on multiple devices.

Camel Labs Performance Testing Framework

Camel IoT Labs comes with the Performance Testing Framework that can be used to define the hardware profiles and test scenarios. Performance framework takes care of detecting the devices connected to your local network, deploying the test application into these, executing the actual tests and generating the results as the human-readable chart. For example the sample output for the MQTT QOS testing could generate the following diagram:

When to use Performance Testing Framework

Performance Testing Framework excels when you would like to answer the following question - how the different field hardware setups perform against the given task. And to answer that question just connect your devices to the local network, execute the performance testing application and compare the generated diagrams.

Here is the 30 seconds video guide demonstrating how easy it is to perform the test:

Hardware profiles

This section covers the hardware profiles for the performance tests. Profiles are used to describe the particular hardware configuration that can be used as a target device for the performance benchmark. Every performance test definition can be executed on the particular hardware profiles.

The example of the hardware profile can be Raspberry PI 2 B+ (aka RPI2).  The RPI2 hardware profile is just the Raspberry Pi 2 B+ model equipped with the network connector (WiFi adapter or the ethernet cable). Currently we assume that the device is running Raspbian operating system (version 2015-05-05).

The other profile we defined for the Camel Labs is the Raspberry is the PI 2 B+ with BU353 (aka RPI2_BU353) The RPI2_BU353 hardware profile is the same as RPI2 profile, but additionally equipped with the BU353 GPS receiver plugged into the USB port.

Running the performance tester 

The easiest way to run the performance benchmark is to connect the target device (for example Rapsberry Pi) into your local network (for example via the WiFi or the Ethernet cable) and start the tester as a Docker container, using the following command:

docker run -v=/tmp/gateway-performance:/tmp/gateway-performance --net=host camellabs/performance-of RPI2

 Keep in mind that RPI2 can be replaced with the other supported hardware profile (like RPI2_BU353). The performance tester detects the tests that can be executed for the given hardware profile, deploys the gateway software to the target device, executes the tests and collects the results. When the execution of the benchmark ends, the result diagrams will be located in the /tmp/gateway-performance directory (or any other directory you specified when executing the command above). The sample diagram may look as follows:

Keep in mind that currently we assume that your Raspberry Pi has default Raspbian SSH account available (username: pi / password: raspberry).

Future of the framework

I plan to develop the gateway Performance Testing Framework to make it the tool for the sizing of the IoT gateways. In particular I plan do add more benchmarks and supported hardware profiles. The performance framework is also a great tool to perform the fully automated end-to-end integration tests on the real hardware - I would definitely like to see the PTF as the tool for running Jenkins-based integration tests executed on the real devices.