Let's start the rhiot!

As many of you probably already know, in the recent months I've been hooked on the Internet Of Thing in general, and creating the rocking Internet Of Things messaging platform in particular. My R&D activities have been revolving around the adoption of the awesome Red Hat JBoss middleware portfolio to the IoT needs. Me, and the other folks interested in the adoption of the projects like Apache Camel or ActiveMQ to the IoT challanges, joined our forces to create the Camel IoT Labs project.

As the Camel IoT Labs project scope is really wider than the scope of the Camel itself, we decided to give up the old project name. Instead we would like to come up with something that would reflect the fact that we are working on the project with the bigger IoT messaging platform picture in mind.

We decided to go with renaming the project to rhiot (pronounced like riot). Our new name hides the Red Hat and IoT in it - which is fine, because we want to stress that our IoT project relies on the rock solid Red Hat software portfolio as its foundation. Starting from today we will be gradually migrating our Camel Labs to new Rhiot name. That will include moving our GitHub project to the new rhiot organization and changing our Maven coordinates from com.github.camellabs to io.rhiot.

I'm also thrilled to say that Red Hat decided to sponsor some R&D activities around the new Rhiot project. So stay tuned and expect amazing IoT stuff coming from the Rhiot team. Also keep in mind that we have really open policy regarding accepting new contributors for our project - if you want to help us to create the IoT messaging platform, you know where to find me :) .

Where am I? Collecting GPS data with Apache Camel

This is the re-post of my article Where Am I? Collecting GPS Data With Apache Camel originally posted at the DZone.

One of the most popular requirement for the field devices used in the IoT systems is to provide the current GPS location of that device and send this information to the data center. In hereby article I will tell you how Apache Camel can turn full-stack Linux microcomputer (like Raspberry Pi) into the device collecting the GPS coordinates.

Which GPS unit should I choose?

There is a myriad of the GPS receivers available in the market. BU353 is one of the most popular and the cheapest GPS units. It can be connected to the computer device via the USB port. If you are looking for good and cheap GPS receiver for your IoT solution, you should definitely consider purchasing this unit.

The picture below presents the BU353 connected to the Raspberry Pi device via the USB port.

You can optionally equip your Pi into the external mobile battery, like GOODRAM Power Bank P661. Equipped with the external power supply you can take your mobile GPS system to the car or into the outdoors. Battery significantly simplifies testing "in the field" part of your solution.

How can Camel help me?

Camel GPS BU353 component can be used to read the current GPS information from that device. With Camel GPS BU353 you can just connect the receiver to your computer's USB port and read the GPS data - the component will make sure that GPS daemon is up, running and switched to the NMEA mode. The component also takes care of parsing the NMEA data read from the serial port, so you can enjoy the GPS data wrapped into the com.github.camellabs.iot.component.gps.bu353.ClientGpsCoordinates POJO objects (which in turn are forwarded to your Camel routes).

Show me the code

In order to take advantage of the Camel GPS BU353 you have to add the following dependency to your Maven project:

    <dependency>
      <groupId>com.github.camel-labs</groupId>
      <artifactId>camel-gps-bu353</artifactId>
      <version>0.1.1</version>
    </dependency>

BU353 component supports only consumer endpoints - it makes sense as GPS component is used to read, not write, GPS data. The BU353 consumer is the polling one, i.e. it periodically asks the GPS device for the current coordinates. The Camel endpoint URI format for the BU353 consumer is as follows:

    gps-bu353:label

Where label can be replaced with any text label:

    from("gps-bu353:current-position").
      to("file:///var/gps-coordinates");

The Camel route presented above reads the current GPS coordinates every 5 seconds and saves these into the /var/gps-coordinates directory. Each GPS coordinates pair is saved into the dedicated file. The complete runnable example of the route above is presented in the code snippet below. I use Camel Spring Boot support to start the Camel context and load the route definition:

@SpringBootApplication
public class GpsReader extends FatJarRouter {

  public void configure() throws Exception {
    from("gps-bu353:current-position").
      to("file:///var/gps-coordinates");
  }

}

BU353 consumer receives the com.github.camellabs.iot.component.gps.bu353.ClientGpsCoordinates instances. ClientGpsCoordinates class is a convenient POJO wrapper around the GPS data:

  ConsumerTemplate consumerTemplate = camelContext.createConsumerTemplate();
  ClientGpsCoordinates currentPosition = 
    consumerTemplate.receiveBody("gps-bu353:current-position", ClientGpsCoordinates.class);

  String deviceId = currentPosition.clientId();
  Date taken = currentPosition.timestamp();
  double latitude = currentPosition.lat();
  double longitude = currentPosition.lng();

ClientGpsCoordinates class name is prefixed with the Client keyword to indicate that these coordinates have been created on the device, not on the server side of the IoT solution.

Process manager

Process manager is used by the BU353 component to execute Linux commands responsible for starting GPSD daemon and configuring the GPS receiver to provide GPS coordinates in the NMEA mode. If for some reason you would like to change the default implementation of the process manager used by Camel (i.e. com.github.camellabs.iot.utils.process.DefaultProcessManager), you can set it on the component level:

    GpsBu353Component bu353 = new GpsBu353Component();
    bu353.setProcessManager(new CustomProcessManager());
    camelContext.addComponent("gps-bu353", bu353);

If the custom process manager is not set on the component, Camel will try to find the manager instance in the registry. So for example for the Spring application, you can just configure the manager as the bean:

    @Bean
    ProcessManager myProcessManager() {
        new CustomProcessManager();
    }

The custom process manager may be useful if for some reasons your Linux distribution requires executing some unusual commands in order to make the GPSD up and running.

What's next?

Do geographical capabilities of the Camel seem compelling to you? Then you should also Camel Geocoder component which can be used to easily convert GPS coordinates collected via Camel into the human readable street address. Also stay tuned for the geofencing features coming soon to the Camel IoT Labs project.

Measuring the throughput of the IoT gateways

The key part of the process of tailoring the perfect IoT solution is choosing the proper hardware for the gateway device. In general the more expensive gateway hardware is, the more messages per second you can process. However the more expensive the gateway device is, the less affordable your IoT solution becomes for the end client. That is the main reason why would you like to have a proper tool for measuring the given IoT messages flow scenario in the unified way, on multiple devices.

Camel Labs Performance Testing Framework

Camel IoT Labs comes with the Performance Testing Framework that can be used to define the hardware profiles and test scenarios. Performance framework takes care of detecting the devices connected to your local network, deploying the test application into these, executing the actual tests and generating the results as the human-readable chart. For example the sample output for the MQTT QOS testing could generate the following diagram:

When to use Performance Testing Framework

Performance Testing Framework excels when you would like to answer the following question - how the different field hardware setups perform against the given task. And to answer that question just connect your devices to the local network, execute the performance testing application and compare the generated diagrams.

Here is the 30 seconds video guide demonstrating how easy it is to perform the test:

Hardware profiles

This section covers the hardware profiles for the performance tests. Profiles are used to describe the particular hardware configuration that can be used as a target device for the performance benchmark. Every performance test definition can be executed on the particular hardware profiles.

The example of the hardware profile can be Raspberry PI 2 B+ (aka RPI2).  The RPI2 hardware profile is just the Raspberry Pi 2 B+ model equipped with the network connector (WiFi adapter or the ethernet cable). Currently we assume that the device is running Raspbian operating system (version 2015-05-05).

The other profile we defined for the Camel Labs is the Raspberry is the PI 2 B+ with BU353 (aka RPI2_BU353) The RPI2_BU353 hardware profile is the same as RPI2 profile, but additionally equipped with the BU353 GPS receiver plugged into the USB port.

Running the performance tester 

The easiest way to run the performance benchmark is to connect the target device (for example Rapsberry Pi) into your local network (for example via the WiFi or the Ethernet cable) and start the tester as a Docker container, using the following command:

docker run -v=/tmp/gateway-performance:/tmp/gateway-performance --net=host camellabs/performance-of RPI2

 Keep in mind that RPI2 can be replaced with the other supported hardware profile (like RPI2_BU353). The performance tester detects the tests that can be executed for the given hardware profile, deploys the gateway software to the target device, executes the tests and collects the results. When the execution of the benchmark ends, the result diagrams will be located in the /tmp/gateway-performance directory (or any other directory you specified when executing the command above). The sample diagram may look as follows:

Keep in mind that currently we assume that your Raspberry Pi has default Raspbian SSH account available (username: pi / password: raspberry).

Future of the framework

I plan to develop the gateway Performance Testing Framework to make it the tool for the sizing of the IoT gateways. In particular I plan do add more benchmarks and supported hardware profiles. The performance framework is also a great tool to perform the fully automated end-to-end integration tests on the real hardware - I would definitely like to see the PTF as the tool for running Jenkins-based integration tests executed on the real devices.

Over-the-air runtime updates of the IoT gateways

This is the repost of my article originally published at the DZone - Over-the-Air Runtime Updates of the IoT Gateways.

The Internet Of Things gateways are usually installed on the devices designed to run for a longer periods of time without the downtime. The is common requirement for the gateway to keep the uninterrupted processing of the messages collected from the field, even when configuration changes have to be applied to the device. While in the Java world the OSGi technology has been commonly identified with the hot redeploy features, I would like to tell you more about the possibilities that Apache Camel running outside the OSGi environment can bring into this topic.

Adding new components at runtime

Apache Camel relies on the components to provide the connectors which can be used to consume messages from the various endpoints (as well as send the messages to the endpoints). For example the gateway based on the Apache Camel could use Paho MQTT component to consume the control commands from the data center and at the same time use the Netty component to communicate with the backed REST services.



Now imagine that at some point we decided to add Salesforce component to our Gateway, so the copy of the message we sent to the backend REST service can be also sent directly to our Salesforce account.



Can we somehow dynamically add the jar containing the Salesforce component to all our production gateway devices deployed into the field? Since the gateways are constantly processing the messages from the sensors, we would like to avoid any restarts of the devices.

Dynamic updates using Camel Grape endpoints

Camel 2.16 comes with new Groovy-based Grape component. Grape is the part of the Groovy language - it is the library that can download the other libraries and add them to the classpath of the current JVM. Grape component downloads jar libraries from the file repositories (including Maven repos) and add these jars dynamically to the application classpath. As soon as the library is downloaded, the Camel can take advantage of new components and data formats provided by these jar files. For example the following Camel route will download and install Camel Saleforce component:

from("direct:start").
  to("grape:org.apache.camel/camel-salesforce/2.15.2");

After the Saleforce jar has been loaded, you can use it in the Camel routes. All that without the restart of the gateway application:

ProducerTemplate template = camelContext.createProducerTemplate();
template.sendBody("direct:dynamicEndpoint", "salesforce:createSObject");
...
from("direct:dynamicEndpoint").
  recipientList().body();

Over-the-air updates

How can we dynamically update many gateways at once using over-the-air technology? As gateways usually run in the environment with the unreliable network connection, I recommend that gateway should listen to the MQTT messages sent from the data center. MQTT client is capable of handling the unstable network connectivity. MQTT client installed on the gateway should connect to the given "update" topic (let's refer to it as over-the-air topic). Whenever we would like to load the component dynamically, we can simply use MQTT over-the-air topic and send the Maven coordinates of the artifact we want to be downloaded to the devices. We can use Camel Paho component for that purpose:

// This code is executed on the data center side.
ProducerTemplate template = camelContext.createProducerTemplate();
// Tell all our devices connected to the over-the-air MQTT topic 
// to download Camel Salesforce connector.
template.sendBody("paho:over-the-air", "org.apache.camel/camel-salesforce/2.15.2");

The following Camel route deployed on the gateway field devices can be used to receive the update requests from the MQTT broker and send them to the Grape component:

from("paho:over-the-air").
  to("grape:over-the-air");

That's all what you need to fetch and load the new library into all of your gateway devices.

Dealing with the download failures

You may be wondering what happens when the Grape component can't download the requested artifact, for example because of the network problems, so common the field environments. The is the moment when Camel and its Camel error handler redelivery policy jumps in. You can easily tell Camel what retry strategy should be used to eventually fetch and load the required dependency. All you have to do is to use a bit of the Camel DSL:

errorHandler(defaultErrorHandler().maximumRedeliveries(10).useExponentialBackOff());

Using this approach you can be sure that Camel will be keep trying to download the requested library. Usually after certain number of attempts to download the artifact, you would like to send back the error MQTT message back to the data center, so your operations team can be notified about the upgrade problems at the given device.

Loading patches on the gateway restart

The another natural question is if the patches loaded by the Grape component will be persisted after the gateway restart. Yes, Camel Grape component keeps track of the patches it loaded (by default the list of the deployed patches is stored in the device file system using plain text file). If you would like to tell Camel to load the installed patches, add the GrapeEndpoint.loadPatches method call into your route definition:

import static org.apache.camel.component.grape.GrapeEndpoint.loadPatches;
...
camelContext.addRoutes(new RouteBuilder() {
  @Override
  public void configure() throws Exception {
    loadPatches(camelContext);
  }
});

Listing patches installed on the gateway

You might be also interested in listing the patches installed on the particular gateway device. The following route demonstrates how to connect embedded HTTP server based on the Netty with the Grape component...

import static org.apache.camel.component.grape.GrapeEndpoint.loadPatches;
...
camelContext.addRoutes(new RouteBuilder() {
  @Override
  public void configure() throws Exception {
    loadPatches(camelContext);

    from("netty4-http:http://0.0.0.0:80/patches").
      setHeader(GrapeConstants.GRAPE_COMMAND, GrapeCommand.listPatches).
      to("grape:listPatches");
  }
});

...so you can open the following URL in the web browser and see the list of the installed artifacts:

$ curl http://your.gateway.com/patches
org.apache.camel/camel-salesforce/2.15.2
org.apache.camel/camel-ftp/2.15.2

Summary

As you can see, Camel in the conjunction with Groovy class loading capabilities provides hot updates features extremely useful for the IoT systems. Camel ability to provide runtime updates over-the-air is an interesting alternative for the OSGi deployments. With the Camel Grape component you can take advantage of the dynamic class loading while keeping all the benefits of the flat classpath.