Monday, May 18, 2015

Make your IoT gateway WiFi-aware using Camel and Kura

This is re-post of my DZone article - Make Your IoT Gateway WiFi-Aware Using Camel and Kura. The original DZone article has been published last week.

The common scenario for the mobile IoT Gateways, for example those mounted on the trucks or the other vehicles, is to cache collected data locally on the device storage and synchronizing the data with the data center only when trusted WiFi access point is available near the gateway. Such trusted WiFi network could be localized near the truck fleet parking.


Using this approach, less urgent data (like GPS coordinates stored for the further offline analysis) can be delivered to the data center without the additional cost related to the GPS transmission fees.



Camel Kura WiFi component from the Camel IoT Labs can be used to retrieve the information about the WiFi access points available within the device range. Under the hood Kura Wifi component uses Kura org.eclipse.kura.net.NetworkService.

Setting up of the Maven project


In order to take advantage of the Camel Kura WiFi component, Maven users should add the following dependency to their POM file:

<dependency>
  <groupId>com.github.camel-labs</groupId>
  <artifactId>camel-kura</artifactId>
  <version>0.0.0</version>
</dependency>

All the other dependencies will be pulled by the Maven transitive resolver.
 

Creating Camel route scanning the WiFi networks


Kura WiFi component supports both the consumer and producer endpoints. In practice it means that you can either periodically scan for the WiFi networks (consumer mode) or explicitly request the single scan (producer mode).

The following Apache Camel route will use the wlan0 interface to scan for the mySsid network. If the mySsid WiFi network will be found, Camel will automatically start the route responsible for the synchronization of the offline data stored on the device local storage with the data center:

  from("kura:wlan0/mySsid").
      to("controlbus:route?routeId=onlineSync&action=start");

  from("file:///var/sensor/temperature").
      routeId("onlineSync").autoStartup(false).
      to("netty4-http://api.mydatacenter.com");
Keep in mind that both network interface and SSID can be replaced with the * wildcards matching respectively all the network interfaces and SSIDs.

For example to read all the SSID available near the device, the following route can be used:

    from("kura:*/*").to(...);


Data returned by the WiFi endpoints


The Kura WiFi consumer returns the list of the org.eclipse.kura.net.wifi.WifiAccessPoint classes returned as a result of the WiFi scan:
    ConsumerTemplate consumerTemplate = camelContext.createConsumerTemplate();
    WifiAccessPoint[] accessPoints = consumerTemplate.receiveBody("kura:wlan0/*", WifiAccessPoint[].class);
You can also request the WiFi scanning using the producer endpoint:
        from("direct:WifiScan").to("kura-wifi:*/*").to("mock:accessPoints");
 
Or using the producer template directly:

    ProducerTemplate template = camelContext.createProducerTemplate();
    WifiAccessPoint[] accessPoints = template.requestBody("kura-wifi:*/*", null, WifiAccessPoint[].class);


Deployment options


Camel Kura WiFi component can be deployed either as the OSGi bundle, directly into the Kura container:

  public class WifiKuraRouter extends KuraRouter {

    @Override
    public void configure() throws Exception {
      from("kura-wifi:*/*").to("log:availableWifiNetworks");
    }

  }

...where KuraRouter is the base OSGi bundle activator for Camel routes deployable into Kura.

The other option of deployment is to use Spring Boot based fat jar:

  @SpringBootApplication
  public class WifiKuraRouter extends FatJar {

    @Override
    public void configure() throws Exception {
      from("kura-wifi:*/*").to("log:availableWifiNetworks");
    }

  }

In the first place Kura Wifi component tries to locate the org.eclipse.kura.net.NetworkService instance in the Camel registry. If exactly one instance of the NetworkService is found (this is usually the case when if you deploy the route into the Kura container), that instance will be used by the Kura component. Otherwise new instance of the org.eclipse.kura.linux.net.NetworkServiceImpl will be created and cached by the KuraAccessPointsProvider.


Summary


Smart WiFi connectivity is fundamental for every mobile IoT gateway solution. Camel Kura WiFi integration makes WiFi scanning process as easy as adding a few lines of the DSL. Your gateway application should try to cache the data collected from the sensors and avoid using the expensive mobile connectivity whenever possible. The efficient IoT gateway should synchronize the stored offline data when the trusted WiFi network is available.

Friday, April 24, 2015

Camel IoT Labs project arrived

As some of you already know, I've been pretty interested in the Internet Of Things topic. The IoT is actually a natural extension of my existing interests - M2M communication relies on the messaging and integration technologies heavily, so Camel, ActiveMQ and Fabric8 excel in this area.

Together with some friends of mine from Red Hat (hello Claus, Rob and Greg!) we started the new umbrella project for the contributions related to the Camel/ActiveMQ and the Internet Of Things. The project is called Camel IoT Labs.





What to expect

In the first release of the project we plan to deliver the following new components for Camel:
  • Raspberry Pi GPIO
  • Raspberry Pi I2C
  • Device IO
  • Tinkerforge
  • Eclipse Kura WiFi

Startup hardware setup

We also plan to create, document and promote something called Camel IoT devKit - the opinionated Raspberry Pi based setup of the hardware that you could use to immediately take advantage of the Camel components deployed into it.

Backend in the cloud

Serious IoT solutions can't exist without the proper data center backend. That's why we also plan to create Cloudlets - customizable microservices based on the Apache Camel providing common backend functionaries required by the IoT systems. Our primary target platform at the moment will be Fabric8 with Openshift 3.

Wanna join us?

That sounds interesting? Then join our efforts! Take a look at our issue tracker - we are open for new ideas. And pull request are always more then welcome :) .

Thursday, March 19, 2015

Camel Paho connector

The incoming version of the Apache Camel (2.16) will bring Paho component which provides connector for the MQTT messaging protocol using the Eclipse Paho library. Paho is one of the most popular MQTT libraries, so if you would like to integrate it with your Java project - Camel Paho connector is a way to go.

How can I use Paho connector?

The basic URI format for the Paho connector is as follows:
paho:queueName[?options]
To me a little more concrete, the following snippet reads messages from the MQTT broker installed on the same host as the Camel router:
from("paho:some/queue").
  to("mock:test");
This one reads messages from the remote MQTT broker:
from("paho:some/queue?brokerUrl=tcp://iot.eclipse.org:1883").
  to("mock:test");
While this sends messages to the MQTT broker:
from("direct:test").
  to("paho:some/target/queue");
The complete Spring Boot based application sending message to the MQTT broker every second, is as simple as:
@SpringBootApplication
class MqttRouter extends FatJarRouter {

    @Override
    void configure() {
        from("timer://trigger").
          setBody().expression { randomUUID().toString() }.
          to("paho:topic?brokerUrl={{broker.url}}")
    }

}

Adding Paho connector to your Maven project

Paho connector is shipped in the dedicated jar which Maven users should add to their pom.xml file:
<dependency>
    <groupid>org.apache.camel</groupid>
    <artifactid>camel-paho</artifactid>
    <version>2.16.0</version>
</dependency>
Keep in mind that Paho artifacts are not hosted in the Maven Central, so you need to add Eclipse Paho repository to your POM xml file as well:
<repositories>
  <repository>
    <id>eclipse-paho</id>
    <url>https://repo.eclipse.org/content/repositories/paho-releases</url>
    <snapshots>
      <enabled>false</enabled>
    </snapshots>
  </repository>
</repositories>

Default payload type

By default Camel Paho component operates on the binary payloads extracted out of (or put into) the MQTT message:
// Receive payload
byte[] payload = (byte[]) consumerTemplate.receiveBody("paho:topic");
 
// Send payload
byte[] payload = "message".getBytes();
producerTemplate.sendBody("paho:topic", payload);
But of course Camel build-in type conversion API can perform the automatic data type transformations for you. In the example below Camel automatically converts binary payload into String (and conversely):
// Receive payload
String payload = consumerTemplate.receiveBody("paho:topic", String.class);
 
// Send payload
String payload = "message";
producerTemplate.sendBody("paho:topic", payload);

Connection options

The convention-over-configuration approach used in Camel is really handy for the most of the situations, but sometimes you would like to have more fine-grained control over the MQTT client connection. To cover such situations just add the bean of type org.eclipse.paho.client.mqttv3.MqttConnectOptions to your Camel registry. For Spring applications that would mean adding bean to your application context. The snippet below uses password-based authentication to connect to the MQTT broker:
@Bean
MqttConnectOptions connectOptions() {
  MqttConnectOptions connectOptions = new MqttConnectOptions();
  connectOptions.setUserName("henry");
  connectOptions.setPassword("secret".toCharArray());
  return connectOptions;
}
That's it. Camel automatically picks up this MqttConnectOptions bean from the registry and use it to establish connection with the MQTT broker.

What's next?

If you are interested in all the available options of the Camel Paho connector, visit the component page. If you are looking for complete example of the project that can be easily deployed into the micro-computers like Raspberry Pi check out CamelM2M MQTT quickstart. If you are wondering how fast MQTT connector can be - check out my previous blog post related to the MQTT performance on Raspberry Pi.

Friday, March 13, 2015

Raspberry Pi 2 and Camel: The MQTT client performance

Raspberry Pi 2 comes armed with the 900 MHz quad-core ARM Cortex-A7 and 1 GB of memory. This is a pretty powerful hardware considering the hobbyist Internet Of Things applications, but it's still relatively slow comparing to the industrial-grade microcomputers. As soon as my very own Raspberry Pi 2 has been shipped to me, I started to wonder how fast this car key size computer can be.


Let's test MQTT client

Together with the friend of mine we decided to create a simple proof of concept demonstrating how fast Raspberry Pi 2 can be in the typical field device or the gateway scenario. The field device scenario is the Raspberry Pi acting as the edge node collecting the information from the sensors...


The gateway scenario is when Raspberry Pi is collecting the messages from the other microcomputers, controllers or sensors...


In both cases RPi is very likely to send the messages to the external MQTT message broker for the further analysis.

Testing conditions

For the purpose of our tests we decided to stick to the first scenario i.e. Raspberry Pi collecting the events and sending them directly to the MQTT broker. We used SSH protocol to put a small Apache Camel application bootstrapped using the Spring Boot to the Pi device. In the Camel router we generated test events using internal Camel timer (emulating the sensor read) and for each event collected we sent a message to the external ActiveMQ MQTT broker. Camel Paho component has been chosen as the MQTT client. In order to decouple events collection from the MQTT sending process, we used in-memory SEDA queue.

What is important, we decided the test the performance of the MQTT at the level 2 of the QOS. Level 2 of MQTT QOS comes with the guaranteed exactly-once message delivery pattern. It provides the highest level of the client reliability, but consumes more client resources (Raspberry Pi processing power and memory in this particular case). 

The broker itself didn't perform any action against the messages it receives. There was no subscriber registered to the MQTT topic we sent messages to. We started dockerized ActiveMQ 5.11 as a message broker (the mentioned Docker image is a part of the Fabric8 project):

docker run -p 1883:1883 -e AMQ_MQTT_PORT=1883 -it fabric8/fabric8-mq:2.0.29

We took the advantage of the Java UUID API to create test messages. Each message was 36-byte long random UUID. Such message size may seem to be small, but events generated by the field sensors are usually not much larger - the majority of the IoT solutions generate the huge number of the small messages.

The application code 

Apache Camel in the conjunction with the Spring Boot creates pretty powerful tool for the M2M solutions. For example to create the application for the purposes of our tests, all the Groovy code we had to write is the snippet presented below:

@SpringBootApplication
class MqttProducerGateway extends FatJarRouter {

    @Override
    void configure() {
        // Read events from the sensors
        from("timer://mockSensor").
                setBody().expression { randomUUID().toString() }.
                to("seda://events") // Enqueue the events in the in-memory queue

        from("seda://events?concurrentConsumers={{broker.consumers:15}}").
                to("paho:topic?brokerUrl={{broker.url}}")
    }

}

That's it! We packaged that code as a fat jar to make deployment via SSH easier. The application is configured from the command line just before the execution of the tests:

java -Dbroker.url=tcp://192.168.1.6 -jar camel-mqtt-benchmark.jar 

First run: 3 consumers sending messages to the MQTT broker

In the first benchmark we used 3 concurrent consumers threads, reading messages from the in-memory SEDA queue and sending those messages to the MQTT broker. Application performs pretty well (up to ~315 messages per second) until Paho client got overwhelmed with the messages produced by the timer. When the number of the messages to be processed became too large, the performance of the gateway started to decrease. The interesting point here is that many messages are produced, while not so many are consumed - that's why we considered increasing the number of the concurrent consumers in the next benchmarking session.


Second run: 15 consumers sending messages to the MQTT broker

We decided to increase the number of the consumers reading messages from the in-memory SEDA queue to 15. After that change route performed really well (up to ~580 messages per second) until Paho client got overwhelmed with the messages produced by the timer. Once again, when the number of the messages to be processed started to be too large, the performance of the gateway started to decrease.

Can we do better?

The biggest problem so far seems to be the fact the Paho slows down when we generate too much messages comparing to the consumption rate. By adding the throttler to the test running 15 consumers, we managed to keep the processing rate on the level of the 700 messages per second. By tuning the consumers settings and reducing the QOS we might increase the number of the messages processed by the Raspberry Pi even more. I will describe how we used Camel throttler to get the stable 700 messages per second in the separated article in the future, as this awesome Camel feature deserves more attention. 

Raspberry Pi 2 results TL;DR;

Raspberry Pi 2 is really fast! As for such small and cheap (35$) device, the performance of the unit is really impressive. You can send almost 700 small QOS 2 messages (36 bytes each) per second from Raspbberry Pi 2 gateway to the MQTT server.

If you plan to run the Paho MQTT client on the RPi 2 remember to:
  • enqueue messages in the internal in-memory queue and use at least 15 concurrent threads to process these messages (as Paho or sensors IO operations may become a bottleneck otherwise)
  • do not let sensors to put too many messages into the queue, otherwise the overall performance of the gateway is decreased significantly. Consider using Camel throttler to limit the number of the messages sent to the queue.

Saturday, February 28, 2015

Apache Camel in the IoT world: Eclipse Kura component

Eclipse Kura is an OSGi based framework dedicated for the M2M gateways based on the small computing platforms like Raspberry Pi or BeagleBoard Black. If you consider using Kura in your M2M gateway and and the same time you would like to take advantage of the rich set of Apache Camel components and its EIP capabilities, then Camel Kura component is something for you. Camel Kura component will be available starting from the Camel 2.15.0.

Camel Kura architecture

The common reason to deploy Camel routes into the Eclipse Kura is to provide enterprise integration patterns and Camel components to the messaging M2M gateway. For example you might want to install Kura on Raspberry PI, then read temperature from the sensor attached to that Raspberry PI using Kura services and finally forward the current temperature value to your data center service using Camel EIP and components. The diagram below demonstrates the architecture of the Camel solution deployed into the Eclipse Kura: 

How can I activate my route?

Bundles deployed to the Eclipse Kura are usually developed as bundle activators. So the easiest way to deploy Apache Camel routes into the Kura is to create an OSGi bundle containing the class extending org.apache.camel.kura.KuraRouterclass:


public class MyKuraRouter extends KuraRouter {
  @Override
  public void configure() throws Exception {
    from("timer:trigger").
      to("netty-http:http://app.mydatacenter.com/api");
  }
}

Keep in mind that KuraRouter implements the org.osgi.framework.BundleActivator interface, so you need to register its start and stop lifecycle methods while creating Kura bundle component class.
Kura router starts its own OSGi-aware CamelContext. It means that for every class extending KuraRouter, there will be a dedicated CamelContext instance. Ideally we recommend to deploy one KuraRouter per OSGi bundle.

How can I deploy my KuraRouter

Bundle containing your Kura router class should import the following packages in the OSGi manifest:

Import-Package: org.osgi.framework;version="1.3.0",
  org.slf4j;version="1.6.4",
  org.apache.camel,org.apache.camel.impl,org.apache.camel.core.osgi,org.apache.camel.builder,org.apache.camel.model,
  org.apache.camel.component.kura

Keep in mind that you don't have to import every Camel component bundle you plan to use in your routes, as Camel components are resolved as the services on the runtime level.
Before you deploy your router bundle, be sure that you have deployed (and started) the following Camel core bundles (using Kura GoGo shell)...

install file:///home/user/.m2/repository/org/apache/camel/camel-core/2.15.0/camel-core-2.15.0.jar
start <camel-core-bundle-id>
install file:///home/user/.m2/repository/org/apache/camel/camel-core-osgi/2.15.0/camel-core-osgi-2.15.0.jar
start <camel-core-osgi-bundle-id>
install file:///home/user/.m2/repository/org/apache/camel/camel-kura/2.15.0/camel-kura-2.15.0.jar
start <camel-kura-bundle-id>

...and all the components you plan to use in your routes:

install file:///home/user/.m2/repository/org/apache/camel/camel-stream/2.15.0/camel-stream-2.15.0.jar
start <camel-stream-bundle-id>

Then finally deploy your router bundle:

install file:///home/user/.m2/repository/com/example/myrouter/1.0/myrouter-1.0.jar
start <your-bundle-id>

Some KuraRouter utilities to make your life easier 

 Kura router base class provides many useful utilities. Let 's explore some of them.


SLF4J logger
Kura uses SLF4J facade for logging purposes. Protected member log returns SLF4J logger instance associated with the given Kura router.

public class MyKuraRouter extends KuraRouter {
    @Override
    public void configure() throws Exception {
        log.info("Configuring Camel routes!");
        ...
    }
}

BundleContext
Protected member bundleContext returns bundle context associated with the given Kura router.

public class MyKuraRouter extends KuraRouter {
    @Override
    public void configure() throws Exception {
        ServiceReference serviceRef = bundleContext.getServiceReference(LogService.class.getName());
        MyService myService = content.getService(serviceRef);
        ...
    }
}

CamelContext
Protected member camelContext is the CamelContext associated with the given Kura router.

public class MyKuraRouter extends KuraRouter {
    @Override
    public void configure() throws Exception {
        camelContext.getStatus();
        ...
    }
}

OSGi service resolver

OSGi service resolver (service(Class serviceType)) can be used to easily retrieve service by type from the OSGi bundle context.

public class MyKuraRouter extends KuraRouter {
    @Override
    public void configure() throws Exception {
        MyService myService = service(MyService.class);
        ...
    }
}

How can I configure CamelContext used by KuraRouter?

Kura router comes with the lifecycle callbacks that can be used to customize the way the Camel router works. For example to configure the CamelContext instance associated with the router just before the former is started, override beforeStartmethod of the KuraRouter class:

public class MyKuraRouter extends KuraRouter {
  ...
  protected void beforeStart(CamelContext camelContext) {
    OsgiDefaultCamelContext osgiContext = (OsgiCamelContext) camelContext;
    osgiContext.setName("NameOfTheRouter");
  }
}

What's next?

The current version of the Camel Kura component provides some useful utilities and simplifies Camel deployments into Kura by providing the opinionated Camel router. However there is still a space for the improvements in the regards of Camel Kura functionality. For example I can imagine predefined Camel components providing consumers/producers for the device services provided by the Kura. Or predefined Camel expressions that could be used to perform the content based routing based on the data read from the Raspberry Pi sensors. I plan to add more features to the Camel Kura in the next releases of Camel. Keep also in mind that we do love contributors in the Apache Camel community  - if you think you got something that could be added to the Camel Kura, just drop me a line!