Thursday, July 30, 2015

Let's start the rhiot!

As many of you probably already know, in the recent months I've been hooked on the Internet Of Thing in general, and creating the rocking Internet Of Things messaging platform in particular. My R&D activities have been revolving around the adoption of the awesome Red Hat JBoss middleware portfolio to the IoT needs. Me, and the other folks interested in the adoption of the projects like Apache Camel or ActiveMQ to the IoT challanges, joined our forces to create the Camel IoT Labs project.

As the Camel IoT Labs project scope is really wider than the scope of the Camel itself, we decided to give up the old project name. Instead we would like to come up with something that would reflect the fact that we are working on the project with the bigger IoT messaging platform picture in mind.

We decided to go with renaming the project to rhiot (pronounced like riot). Our new name hides the Red Hat and IoT in it - which is fine, because we want to stress that our IoT project relies on the rock solid Red Hat software portfolio as its foundation. Starting from today we will be gradually migrating our Camel Labs to new Rhiot name. That will include moving our GitHub project to the new rhiot organization and changing our Maven coordinates from com.github.camellabs to io.rhiot.

I'm also thrilled to say that Red Hat decided to sponsor some R&D activities around the new Rhiot project. So stay tuned and expect amazing IoT stuff coming from the Rhiot team. Also keep in mind that we have really open policy regarding accepting new contributors for our project - if you want to help us to create the IoT messaging platform, you know where to find me :) .

Wednesday, July 29, 2015

Where am I? Collecting GPS data with Apache Camel

This is the re-post of my article Where Am I? Collecting GPS Data With Apache Camel originally posted at the DZone.

One of the most popular requirement for the field devices used in the IoT systems is to provide the current GPS location of that device and send this information to the data center. In hereby article I will tell you how Apache Camel can turn full-stack Linux microcomputer (like Raspberry Pi) into the device collecting the GPS coordinates.

Which GPS unit should I choose?

There is a myriad of the GPS receivers available in the market. BU353 is one of the most popular and the cheapest GPS units. It can be connected to the computer device via the USB port. If you are looking for good and cheap GPS receiver for your IoT solution, you should definitely consider purchasing this unit.

The picture below presents the BU353 connected to the Raspberry Pi device via the USB port.
You can optionally equip your Pi into the external mobile battery, like GOODRAM Power Bank P661. Equipped with the external power supply you can take your mobile GPS system to the car or into the outdoors. Battery significantly simplifies testing "in the field" part of your solution.


How can Camel help me?

Camel GPS BU353 component can be used to read the current GPS information from that device. With Camel GPS BU353 you can just connect the receiver to your computer's USB port and read the GPS data - the component will make sure that GPS daemon is up, running and switched to the NMEA mode. The component also takes care of parsing the NMEA data read from the serial port, so you can enjoy the GPS data wrapped into the com.github.camellabs.iot.component.gps.bu353.ClientGpsCoordinates POJO objects (which in turn are forwarded to your Camel routes).

Show me the code

In order to take advantage of the Camel GPS BU353 you have to add the following dependency to your Maven project:
    <dependency>
      <groupId>com.github.camel-labs</groupId>
      <artifactId>camel-gps-bu353</artifactId>
      <version>0.1.1</version>
    </dependency>
BU353 component supports only consumer endpoints - it makes sense as GPS component is used to read, not write, GPS data. The BU353 consumer is the polling one, i.e. it periodically asks the GPS device for the current coordinates. The Camel endpoint URI format for the BU353 consumer is as follows:
    gps-bu353:label
Where label can be replaced with any text label:
    from("gps-bu353:current-position").
      to("file:///var/gps-coordinates");
The Camel route presented above reads the current GPS coordinates every 5 seconds and saves these into the /var/gps-coordinates directory. Each GPS coordinates pair is saved into the dedicated file. The complete runnable example of the route above is presented in the code snippet below. I use Camel Spring Boot support to start the Camel context and load the route definition:
@SpringBootApplication
public class GpsReader extends FatJarRouter {

  public void configure() throws Exception {
    from("gps-bu353:current-position").
      to("file:///var/gps-coordinates");
  }

}
BU353 consumer receives the com.github.camellabs.iot.component.gps.bu353.ClientGpsCoordinates instances. ClientGpsCoordinates class is a convenient POJO wrapper around the GPS data:
  ConsumerTemplate consumerTemplate = camelContext.createConsumerTemplate();
  ClientGpsCoordinates currentPosition = 
    consumerTemplate.receiveBody("gps-bu353:current-position", ClientGpsCoordinates.class);

  String deviceId = currentPosition.clientId();
  Date taken = currentPosition.timestamp();
  double latitude = currentPosition.lat();
  double longitude = currentPosition.lng();
ClientGpsCoordinates class name is prefixed with the Client keyword to indicate that these coordinates have been created on the device, not on the server side of the IoT solution.

Process manager

Process manager is used by the BU353 component to execute Linux commands responsible for starting GPSD daemon and configuring the GPS receiver to provide GPS coordinates in the NMEA mode. If for some reason you would like to change the default implementation of the process manager used by Camel (i.e. com.github.camellabs.iot.utils.process.DefaultProcessManager), you can set it on the component level:
    GpsBu353Component bu353 = new GpsBu353Component();
    bu353.setProcessManager(new CustomProcessManager());
    camelContext.addComponent("gps-bu353", bu353);
If the custom process manager is not set on the component, Camel will try to find the manager instance in the registry. So for example for the Spring application, you can just configure the manager as the bean:
    @Bean
    ProcessManager myProcessManager() {
        new CustomProcessManager();
    }
The custom process manager may be useful if for some reasons your Linux distribution requires executing some unusual commands in order to make the GPSD up and running.

What's next?

Do geographical capabilities of the Camel seem compelling to you? Then you should also Camel Geocoder component which can be used to easily convert GPS coordinates collected via Camel into the human readable street address. Also stay tuned for the geofencing features coming soon to the Camel IoT Labs project.

Thursday, July 23, 2015

Measuring the throughput of the IoT gateways

The key part of the process of tailoring the perfect IoT solution is choosing the proper hardware for the gateway device. In general the more expensive gateway hardware is, the more messages per second you can process. However the more expensive the gateway device is, the less affordable your IoT solution becomes for the end client. That is the main reason why would you like to have a proper tool for measuring the given IoT messages flow scenario in the unified way, on multiple devices.


Camel Labs Performance Testing Framework

Camel IoT Labs comes with the Performance Testing Framework that can be used to define the hardware profiles and test scenarios. Performance framework takes care of detecting the devices connected to your local network, deploying the test application into these, executing the actual tests and generating the results as the human-readable chart. For example the sample output for the MQTT QOS testing could generate the following diagram:



When to use Performance Testing Framework

Performance Testing Framework excels when you would like to answer the following question - how the different field hardware setups perform against the given task. And to answer that question just connect your devices to the local network, execute the performance testing application and compare the generated diagrams.

Here is the 30 seconds video guide demonstrating how easy it is to perform the test:




Hardware profiles

This section covers the hardware profiles for the performance tests. Profiles are used to describe the particular hardware configuration that can be used as a target device for the performance benchmark. Every performance test definition can be executed on the particular hardware profiles.

The example of the hardware profile can be Raspberry PI 2 B+ (aka RPI2).  The RPI2 hardware profile is just the Raspberry Pi 2 B+ model equipped with the network connector (WiFi adapter or the ethernet cable). Currently we assume that the device is running Raspbian operating system (version 2015-05-05).






The other profile we defined for the Camel Labs is the Raspberry is the PI 2 B+ with BU353 (aka RPI2_BU353) The RPI2_BU353 hardware profile is the same as RPI2 profile, but additionally equipped with the BU353 GPS receiver plugged into the USB port.




Running the performance tester 

The easiest way to run the performance benchmark is to connect the target device (for example Rapsberry Pi) into your local network (for example via the WiFi or the Ethernet cable) and start the tester as a Docker container, using the following command:

docker run -v=/tmp/gateway-performance:/tmp/gateway-performance --net=host camellabs/performance-of RPI2

 Keep in mind that RPI2 can be replaced with the other supported hardware profile (like RPI2_BU353). The performance tester detects the tests that can be executed for the given hardware profile, deploys the gateway software to the target device, executes the tests and collects the results. When the execution of the benchmark ends, the result diagrams will be located in the /tmp/gateway-performance directory (or any other directory you specified when executing the command above). The sample diagram may look as follows:




Keep in mind that currently we assume that your Raspberry Pi has default Raspbian SSH account available (username: pi / password: raspberry).

Future of the framework

I plan to develop the gateway Performance Testing Framework to make it the tool for the sizing of the IoT gateways. In particular I plan do add more benchmarks and supported hardware profiles. The performance framework is also a great tool to perform the fully automated end-to-end integration tests on the real hardware - I would definitely like to see the PTF as the tool for running Jenkins-based integration tests executed on the real devices.

Over-the-air runtime updates of the IoT gateways

This is the repost of my article originally published at the DZone - Over-the-Air Runtime Updates of the IoT Gateways.

The Internet Of Things gateways are usually installed on the devices designed to run for a longer periods of time without the downtime. The is common requirement for the gateway to keep the uninterrupted processing of the messages collected from the field, even when configuration changes have to be applied to the device. While in the Java world the OSGi technology has been commonly identified with the hot redeploy features, I would like to tell you more about the possibilities that Apache Camel running outside the OSGi environment can bring into this topic.


Adding new components at runtime


Apache Camel relies on the components to provide the connectors which can be used to consume messages from the various endpoints (as well as send the messages to the endpoints). For example the gateway based on the Apache Camel could use Paho MQTT component to consume the control commands from the data center and at the same time use the Netty component to communicate with the backed REST services.



Now imagine that at some point we decided to add Salesforce component to our Gateway, so the copy of the message we sent to the backend REST service can be also sent directly to our Salesforce account.



Can we somehow dynamically add the jar containing the Salesforce component to all our production gateway devices deployed into the field? Since the gateways are constantly processing the messages from the sensors, we would like to avoid any restarts of the devices.



Dynamic updates using Camel Grape endpoints


Camel 2.16 comes with new Groovy-based Grape component. Grape is the part of the Groovy language - it is the library that can download the other libraries and add them to the classpath of the current JVM. Grape component downloads jar libraries from the file repositories (including Maven repos) and add these jars dynamically to the application classpath. As soon as the library is downloaded, the Camel can take advantage of new components and data formats provided by these jar files. For example the following Camel route will download and install Camel Saleforce component:
from("direct:start").
  to("grape:org.apache.camel/camel-salesforce/2.15.2");

After the Saleforce jar has been loaded, you can use it in the Camel routes. All that without the restart of the gateway application:

ProducerTemplate template = camelContext.createProducerTemplate();
template.sendBody("direct:dynamicEndpoint", "salesforce:createSObject");
...
from("direct:dynamicEndpoint").
  recipientList().body();



Over-the-air updates


How can we dynamically update many gateways at once using over-the-air technology? As gateways usually run in the environment with the unreliable network connection, I recommend that gateway should listen to the MQTT messages sent from the data center. MQTT client is capable of handling the unstable network connectivity. MQTT client installed on the gateway should connect to the given "update" topic (let's refer to it as over-the-air topic). Whenever we would like to load the component dynamically, we can simply use MQTT over-the-air topic and send the Maven coordinates of the artifact we want to be downloaded to the devices. We can use Camel Paho component for that purpose:
// This code is executed on the data center side.
ProducerTemplate template = camelContext.createProducerTemplate();
// Tell all our devices connected to the over-the-air MQTT topic 
// to download Camel Salesforce connector.
template.sendBody("paho:over-the-air", "org.apache.camel/camel-salesforce/2.15.2");

The following Camel route deployed on the gateway field devices can be used to receive the update requests from the MQTT broker and send them to the Grape component:
from("paho:over-the-air").
  to("grape:over-the-air");

That's all what you need to fetch and load the new library into all of your gateway devices.


Dealing with the download failures


You may be wondering what happens when the Grape component can't download the requested artifact, for example because of the network problems, so common the field environments. The is the moment when Camel and its Camel error handler redelivery policy jumps in. You can easily tell Camel what retry strategy should be used to eventually fetch and load the required dependency. All you have to do is to use a bit of the Camel DSL:
errorHandler(defaultErrorHandler().maximumRedeliveries(10).useExponentialBackOff());

Using this approach you can be sure that Camel will be keep trying to download the requested library. Usually after certain number of attempts to download the artifact, you would like to send back the error MQTT message back to the data center, so your operations team can be notified about the upgrade problems at the given device.


Loading patches on the gateway restart


The another natural question is if the patches loaded by the Grape component will be persisted after the gateway restart. Yes, Camel Grape component keeps track of the patches it loaded (by default the list of the deployed patches is stored in the device file system using plain text file). If you would like to tell Camel to load the installed patches, add the GrapeEndpoint.loadPatches method call into your route definition:
import static org.apache.camel.component.grape.GrapeEndpoint.loadPatches;
...
camelContext.addRoutes(new RouteBuilder() {
  @Override
  public void configure() throws Exception {
    loadPatches(camelContext);
  }
});


Listing patches installed on the gateway


You might be also interested in listing the patches installed on the particular gateway device. The following route demonstrates how to connect embedded HTTP server based on the Netty with the Grape component...

import static org.apache.camel.component.grape.GrapeEndpoint.loadPatches;
...
camelContext.addRoutes(new RouteBuilder() {
  @Override
  public void configure() throws Exception {
    loadPatches(camelContext);

    from("netty4-http:http://0.0.0.0:80/patches").
      setHeader(GrapeConstants.GRAPE_COMMAND, GrapeCommand.listPatches).
      to("grape:listPatches");
  }
});


...so you can open the following URL in the web browser and see the list of the installed artifacts:

$ curl http://your.gateway.com/patches
org.apache.camel/camel-salesforce/2.15.2
org.apache.camel/camel-ftp/2.15.2


Summary


As you can see, Camel in the conjunction with Groovy class loading capabilities provides hot updates features extremely useful for the IoT systems. Camel ability to provide runtime updates over-the-air is an interesting alternative for the OSGi deployments. With the Camel Grape component you can take advantage of the dynamic class loading while keeping all the benefits of the flat classpath.

Monday, May 18, 2015

Make your IoT gateway WiFi-aware using Camel and Kura

This is re-post of my DZone article - Make Your IoT Gateway WiFi-Aware Using Camel and Kura. The original DZone article has been published last week.

The common scenario for the mobile IoT Gateways, for example those mounted on the trucks or the other vehicles, is to cache collected data locally on the device storage and synchronizing the data with the data center only when trusted WiFi access point is available near the gateway. Such trusted WiFi network could be localized near the truck fleet parking.


Using this approach, less urgent data (like GPS coordinates stored for the further offline analysis) can be delivered to the data center without the additional cost related to the GPS transmission fees.



Camel Kura WiFi component from the Camel IoT Labs can be used to retrieve the information about the WiFi access points available within the device range. Under the hood Kura Wifi component uses Kura org.eclipse.kura.net.NetworkService.

Setting up of the Maven project


In order to take advantage of the Camel Kura WiFi component, Maven users should add the following dependency to their POM file:

<dependency>
  <groupId>com.github.camel-labs</groupId>
  <artifactId>camel-kura</artifactId>
  <version>0.0.0</version>
</dependency>

All the other dependencies will be pulled by the Maven transitive resolver.
 

Creating Camel route scanning the WiFi networks


Kura WiFi component supports both the consumer and producer endpoints. In practice it means that you can either periodically scan for the WiFi networks (consumer mode) or explicitly request the single scan (producer mode).

The following Apache Camel route will use the wlan0 interface to scan for the mySsid network. If the mySsid WiFi network will be found, Camel will automatically start the route responsible for the synchronization of the offline data stored on the device local storage with the data center:

  from("kura:wlan0/mySsid").
      to("controlbus:route?routeId=onlineSync&action=start");

  from("file:///var/sensor/temperature").
      routeId("onlineSync").autoStartup(false).
      to("netty4-http://api.mydatacenter.com");
Keep in mind that both network interface and SSID can be replaced with the * wildcards matching respectively all the network interfaces and SSIDs.

For example to read all the SSID available near the device, the following route can be used:

    from("kura:*/*").to(...);


Data returned by the WiFi endpoints


The Kura WiFi consumer returns the list of the org.eclipse.kura.net.wifi.WifiAccessPoint classes returned as a result of the WiFi scan:
    ConsumerTemplate consumerTemplate = camelContext.createConsumerTemplate();
    WifiAccessPoint[] accessPoints = consumerTemplate.receiveBody("kura:wlan0/*", WifiAccessPoint[].class);
You can also request the WiFi scanning using the producer endpoint:
        from("direct:WifiScan").to("kura-wifi:*/*").to("mock:accessPoints");
 
Or using the producer template directly:

    ProducerTemplate template = camelContext.createProducerTemplate();
    WifiAccessPoint[] accessPoints = template.requestBody("kura-wifi:*/*", null, WifiAccessPoint[].class);


Deployment options


Camel Kura WiFi component can be deployed either as the OSGi bundle, directly into the Kura container:

  public class WifiKuraRouter extends KuraRouter {

    @Override
    public void configure() throws Exception {
      from("kura-wifi:*/*").to("log:availableWifiNetworks");
    }

  }

...where KuraRouter is the base OSGi bundle activator for Camel routes deployable into Kura.

The other option of deployment is to use Spring Boot based fat jar:

  @SpringBootApplication
  public class WifiKuraRouter extends FatJar {

    @Override
    public void configure() throws Exception {
      from("kura-wifi:*/*").to("log:availableWifiNetworks");
    }

  }

In the first place Kura Wifi component tries to locate the org.eclipse.kura.net.NetworkService instance in the Camel registry. If exactly one instance of the NetworkService is found (this is usually the case when if you deploy the route into the Kura container), that instance will be used by the Kura component. Otherwise new instance of the org.eclipse.kura.linux.net.NetworkServiceImpl will be created and cached by the KuraAccessPointsProvider.


Summary


Smart WiFi connectivity is fundamental for every mobile IoT gateway solution. Camel Kura WiFi integration makes WiFi scanning process as easy as adding a few lines of the DSL. Your gateway application should try to cache the data collected from the sensors and avoid using the expensive mobile connectivity whenever possible. The efficient IoT gateway should synchronize the stored offline data when the trusted WiFi network is available.

Friday, April 24, 2015

Camel IoT Labs project arrived

As some of you already know, I've been pretty interested in the Internet Of Things topic. The IoT is actually a natural extension of my existing interests - M2M communication relies on the messaging and integration technologies heavily, so Camel, ActiveMQ and Fabric8 excel in this area.

Together with some friends of mine from Red Hat (hello Claus, Rob and Greg!) we started the new umbrella project for the contributions related to the Camel/ActiveMQ and the Internet Of Things. The project is called Camel IoT Labs.





What to expect

In the first release of the project we plan to deliver the following new components for Camel:
  • Raspberry Pi GPIO
  • Raspberry Pi I2C
  • Device IO
  • Tinkerforge
  • Eclipse Kura WiFi

Startup hardware setup

We also plan to create, document and promote something called Camel IoT devKit - the opinionated Raspberry Pi based setup of the hardware that you could use to immediately take advantage of the Camel components deployed into it.

Backend in the cloud

Serious IoT solutions can't exist without the proper data center backend. That's why we also plan to create Cloudlets - customizable microservices based on the Apache Camel providing common backend functionaries required by the IoT systems. Our primary target platform at the moment will be Fabric8 with Openshift 3.

Wanna join us?

That sounds interesting? Then join our efforts! Take a look at our issue tracker - we are open for new ideas. And pull request are always more then welcome :) .

Thursday, March 19, 2015

Camel Paho connector

The incoming version of the Apache Camel (2.16) will bring Paho component which provides connector for the MQTT messaging protocol using the Eclipse Paho library. Paho is one of the most popular MQTT libraries, so if you would like to integrate it with your Java project - Camel Paho connector is a way to go.

How can I use Paho connector?

The basic URI format for the Paho connector is as follows:
paho:queueName[?options]
To me a little more concrete, the following snippet reads messages from the MQTT broker installed on the same host as the Camel router:
from("paho:some/queue").
  to("mock:test");
This one reads messages from the remote MQTT broker:
from("paho:some/queue?brokerUrl=tcp://iot.eclipse.org:1883").
  to("mock:test");
While this sends messages to the MQTT broker:
from("direct:test").
  to("paho:some/target/queue");
The complete Spring Boot based application sending message to the MQTT broker every second, is as simple as:
@SpringBootApplication
class MqttRouter extends FatJarRouter {

    @Override
    void configure() {
        from("timer://trigger").
          setBody().expression { randomUUID().toString() }.
          to("paho:topic?brokerUrl={{broker.url}}")
    }

}

Adding Paho connector to your Maven project

Paho connector is shipped in the dedicated jar which Maven users should add to their pom.xml file:
<dependency>
    <groupid>org.apache.camel</groupid>
    <artifactid>camel-paho</artifactid>
    <version>2.16.0</version>
</dependency>
Keep in mind that Paho artifacts are not hosted in the Maven Central, so you need to add Eclipse Paho repository to your POM xml file as well:
<repositories>
  <repository>
    <id>eclipse-paho</id>
    <url>https://repo.eclipse.org/content/repositories/paho-releases</url>
    <snapshots>
      <enabled>false</enabled>
    </snapshots>
  </repository>
</repositories>

Default payload type

By default Camel Paho component operates on the binary payloads extracted out of (or put into) the MQTT message:
// Receive payload
byte[] payload = (byte[]) consumerTemplate.receiveBody("paho:topic");
 
// Send payload
byte[] payload = "message".getBytes();
producerTemplate.sendBody("paho:topic", payload);
But of course Camel build-in type conversion API can perform the automatic data type transformations for you. In the example below Camel automatically converts binary payload into String (and conversely):
// Receive payload
String payload = consumerTemplate.receiveBody("paho:topic", String.class);
 
// Send payload
String payload = "message";
producerTemplate.sendBody("paho:topic", payload);

Connection options

The convention-over-configuration approach used in Camel is really handy for the most of the situations, but sometimes you would like to have more fine-grained control over the MQTT client connection. To cover such situations just add the bean of type org.eclipse.paho.client.mqttv3.MqttConnectOptions to your Camel registry. For Spring applications that would mean adding bean to your application context. The snippet below uses password-based authentication to connect to the MQTT broker:
@Bean
MqttConnectOptions connectOptions() {
  MqttConnectOptions connectOptions = new MqttConnectOptions();
  connectOptions.setUserName("henry");
  connectOptions.setPassword("secret".toCharArray());
  return connectOptions;
}
That's it. Camel automatically picks up this MqttConnectOptions bean from the registry and use it to establish connection with the MQTT broker.

What's next?

If you are interested in all the available options of the Camel Paho connector, visit the component page. If you are looking for complete example of the project that can be easily deployed into the micro-computers like Raspberry Pi check out CamelM2M MQTT quickstart. If you are wondering how fast MQTT connector can be - check out my previous blog post related to the MQTT performance on Raspberry Pi.