Showing posts with label ActiveMQ. Show all posts
Showing posts with label ActiveMQ. Show all posts

Thursday, November 5, 2015

IoT AMQP backend in seconds (with Rhiot)

This is repost of my DZone article - Building an IoT AMQP Backend in Seconds With Rhiot.
The AMQP is becoming widely adopted as the protocol of choice the communication between an IoT gateway and a data center. If you would like to rapidly create the AMQP backend service that can be immediately ready for your gateways and devices, the Rhiot AMQP quickstart will be more than interesting for you.

The AMQP cloudlet quickstart can be used as a base for the fat-jar AMQP microservices (aka cloudlets). If you wanna create a simple backend application capable of exposing AMQP-endpoint and handling the AMQP-based communication, the AMQT cloudlet quickstart is the best way to start your development efforts.

Creating and running the AMQP cloudlet project

In order to create the AMQP cloudlet project execute the following commands:
git clone git@github.com:rhiot/quickstarts.git
cp -r quickstarts/cloudlets/amqp amqp
cd amqp
mvn install
To start the AMQP cloudlet execute the following command:
java -jar target/rhiot-cloudlets-amqp-1.0.0-SNAPSHOT.jar
That is really all you need to expose the AMQP message broker to the external devices and gateways.
You can also build and run it as a Docker image (we love Docker and highly recommend this approach):
TARGET_IMAGE=yourUsername/rhiot-cloudlets-amqp
mvn install docker:build docker:push -Ddocker.image.target=${TARGET_IMAGE}
docker run -it ${TARGET_IMAGE}

AMQP broker

By default AMQP cloudlet quickstart starts embedded ActiveMQ AMQP broker on 5672 port. If you would like to connect your cloudlet application to the external ActiveMQ broker (instead of starting the embedded one), run the cloudlet with the BROKER_URL environment variable or system property, for example:
java -DBROKER_URL=tcp://amqbroker.example.com:61616 -jar target/rhiot-cloudlets-amqp-1.0.0-SNAPSHOT.jar
...or...
docker run -e BROKER_URL=tcp://amqbroker.example.com:61616 -it yourUsername/rhiot-cloudlets-amqp

Sample chat application

The AMQP cloudlet quickstart is in fact a simple chat application. Clients can send the messages to the chat channel by subscribing to the broker and sending the messages to the chat AMQP queue.
$ amqp-client -h localhost -p 5672 --topic chat "Hello, this is the IoT device!"

The clients can subscribe to the chat updates by listening on the chat-updates AMQP topic - whenever the new message has been sent to the chat channel, the clients registered to the chat-updates will receive the updated chat history.
$ amqp-client -h localhost -p 5672 --subscribe --topic chat-updates
Hello, this is the IoT device!
I just wanted to say hello!
Hello, IoT device. Nice to meet you!

The quickstart also exposes the simple REST API that can be used to read the chat history using the HTTP GET request:
$ curl http://localhost:8180/chat
Hello, this is the IoT device!
I just wanted to say hello!
Hello, IoT device. Nice to meet you!

Architectural overview

When AMQP cloudlet is started with the embedded ActiveMQ broker, the architecture of the example is the following:
When you connect to the external ActiveMQ broker (using BROKER_URL option), the architecture of the example becomes more like the following diagram:

The quickstart code

You may be wondering how much code do you need in order to take the advantage of the presented AMQP functionality. Below is all code used by our quickstart to handle the AMQP connectivity: 


import io.rhiot.steroids.camel.CamelBootstrap;

import java.util.LinkedList;
import java.util.List;

import static io.rhiot.steroids.activemq.EmbeddedActiveMqBrokerBootInitializer.amqpJmsBridge;
import static org.apache.commons.lang3.StringUtils.join;

public class ChatCloudlet extends CamelBootstrap {

    static final List chat = new LinkedList<>();

    @Override
    public void configure() throws Exception {
        from(amqpJmsBridge("chat")).process(
                exchange -> chat.add(exchange.getIn().getBody(String.class))
        ).process(
                exchange -> exchange.getIn().setBody(join(chat, "\n"))
        ).to(amqpJmsBridge("topic:chat-updates"));
    }

}

And below is the code used to start the REST API mentioned before:

import io.rhiot.steroids.camel.Route;
import org.apache.camel.builder.RouteBuilder;

import static org.apache.commons.lang3.StringUtils.join;

@Route
public class ChatRestApiRoutes extends RouteBuilder {

    @Override
    public void configure() throws Exception {
        restConfiguration().component("netty4-http").host("0.0.0.0").port(8180);

        rest("/chat").get().route().process(
                exchange -> exchange.getIn().setBody(join(ChatCloudlet.chat, "\n"))
        );
    }

}

As you can see, Apache Camel is the first class citizen in the Rhiot world. In order to make Camel connectivity easier, Rhiot comes with the static DSL methods like  EmbeddedActiveMqBrokerBootInitializer.amqpJmsBridge which can be used to easily create the Camel endpoints associated with the broker used by the quickstart.

Friday, April 24, 2015

Camel IoT Labs project arrived

As some of you already know, I've been pretty interested in the Internet Of Things topic. The IoT is actually a natural extension of my existing interests - M2M communication relies on the messaging and integration technologies heavily, so Camel, ActiveMQ and Fabric8 excel in this area.

Together with some friends of mine from Red Hat (hello Claus, Rob and Greg!) we started the new umbrella project for the contributions related to the Camel/ActiveMQ and the Internet Of Things. The project is called Camel IoT Labs.

What to expect

In the first release of the project we plan to deliver the following new components for Camel:
  • Raspberry Pi GPIO
  • Raspberry Pi I2C
  • Device IO
  • Tinkerforge
  • Eclipse Kura WiFi

Startup hardware setup

We also plan to create, document and promote something called Camel IoT devKit - the opinionated Raspberry Pi based setup of the hardware that you could use to immediately take advantage of the Camel components deployed into it.

Backend in the cloud

Serious IoT solutions can't exist without the proper data center backend. That's why we also plan to create Cloudlets - customizable microservices based on the Apache Camel providing common backend functionaries required by the IoT systems. Our primary target platform at the moment will be Fabric8 with Openshift 3.

Wanna join us?

That sounds interesting? Then join our efforts! Take a look at our issue tracker - we are open for new ideas. And pull request are always more then welcome :) .

Friday, March 13, 2015

Raspberry Pi 2 and Camel: The MQTT client performance

Raspberry Pi 2 comes armed with the 900 MHz quad-core ARM Cortex-A7 and 1 GB of memory. This is a pretty powerful hardware considering the hobbyist Internet Of Things applications, but it's still relatively slow comparing to the industrial-grade microcomputers. As soon as my very own Raspberry Pi 2 has been shipped to me, I started to wonder how fast this car key size computer can be.


Let's test MQTT client

Together with the friend of mine we decided to create a simple proof of concept demonstrating how fast Raspberry Pi 2 can be in the typical field device or the gateway scenario. The field device scenario is the Raspberry Pi acting as the edge node collecting the information from the sensors...


The gateway scenario is when Raspberry Pi is collecting the messages from the other microcomputers, controllers or sensors...


In both cases RPi is very likely to send the messages to the external MQTT message broker for the further analysis.

Testing conditions

For the purpose of our tests we decided to stick to the first scenario i.e. Raspberry Pi collecting the events and sending them directly to the MQTT broker. We used SSH protocol to put a small Apache Camel application bootstrapped using the Spring Boot to the Pi device. In the Camel router we generated test events using internal Camel timer (emulating the sensor read) and for each event collected we sent a message to the external ActiveMQ MQTT broker. Camel Paho component has been chosen as the MQTT client. In order to decouple events collection from the MQTT sending process, we used in-memory SEDA queue.

What is important, we decided the test the performance of the MQTT at the level 2 of the QOS. Level 2 of MQTT QOS comes with the guaranteed exactly-once message delivery pattern. It provides the highest level of the client reliability, but consumes more client resources (Raspberry Pi processing power and memory in this particular case). 

The broker itself didn't perform any action against the messages it receives. There was no subscriber registered to the MQTT topic we sent messages to. We started dockerized ActiveMQ 5.11 as a message broker (the mentioned Docker image is a part of the Fabric8 project):

docker run -p 1883:1883 -e AMQ_MQTT_PORT=1883 -it fabric8/fabric8-mq:2.0.29

We took the advantage of the Java UUID API to create test messages. Each message was 36-byte long random UUID. Such message size may seem to be small, but events generated by the field sensors are usually not much larger - the majority of the IoT solutions generate the huge number of the small messages.

The application code 

Apache Camel in the conjunction with the Spring Boot creates pretty powerful tool for the M2M solutions. For example to create the application for the purposes of our tests, all the Groovy code we had to write is the snippet presented below:

@SpringBootApplication
class MqttProducerGateway extends FatJarRouter {

    @Override
    void configure() {
        // Read events from the sensors
        from("timer://mockSensor").
                setBody().expression { randomUUID().toString() }.
                to("seda://events") // Enqueue the events in the in-memory queue

        from("seda://events?concurrentConsumers={{broker.consumers:15}}").
                to("paho:topic?brokerUrl={{broker.url}}")
    }

}

That's it! We packaged that code as a fat jar to make deployment via SSH easier. The application is configured from the command line just before the execution of the tests:

java -Dbroker.url=tcp://192.168.1.6 -jar camel-mqtt-benchmark.jar 

First run: 3 consumers sending messages to the MQTT broker

In the first benchmark we used 3 concurrent consumers threads, reading messages from the in-memory SEDA queue and sending those messages to the MQTT broker. Application performs pretty well (up to ~315 messages per second) until Paho client got overwhelmed with the messages produced by the timer. When the number of the messages to be processed became too large, the performance of the gateway started to decrease. The interesting point here is that many messages are produced, while not so many are consumed - that's why we considered increasing the number of the concurrent consumers in the next benchmarking session.


Second run: 15 consumers sending messages to the MQTT broker

We decided to increase the number of the consumers reading messages from the in-memory SEDA queue to 15. After that change route performed really well (up to ~580 messages per second) until Paho client got overwhelmed with the messages produced by the timer. Once again, when the number of the messages to be processed started to be too large, the performance of the gateway started to decrease.

Can we do better?

The biggest problem so far seems to be the fact the Paho slows down when we generate too much messages comparing to the consumption rate. By adding the throttler to the test running 15 consumers, we managed to keep the processing rate on the level of the 700 messages per second. By tuning the consumers settings and reducing the QOS we might increase the number of the messages processed by the Raspberry Pi even more. I will describe how we used Camel throttler to get the stable 700 messages per second in the separated article in the future, as this awesome Camel feature deserves more attention. 

Raspberry Pi 2 results TL;DR;

Raspberry Pi 2 is really fast! As for such small and cheap (35$) device, the performance of the unit is really impressive. You can send almost 700 small QOS 2 messages (36 bytes each) per second from Raspbberry Pi 2 gateway to the MQTT server.

If you plan to run the Paho MQTT client on the RPi 2 remember to:
  • enqueue messages in the internal in-memory queue and use at least 15 concurrent threads to process these messages (as Paho or sensors IO operations may become a bottleneck otherwise)
  • do not let sensors to put too many messages into the queue, otherwise the overall performance of the gateway is decreased significantly. Consider using Camel throttler to limit the number of the messages sent to the queue.

Thursday, August 30, 2012

ServiceMix on the bleeding edge

Why stable dependencies are cool

ServiceMix (and Fuse ESB as well) are supposed to work with a specific version of Camel. For example ServiceMix 4.4.2 comes with a pre-configured Camel 2.8.5 features repository.
karaf@root> features:listurl | grep camel
  true    mvn:org.apache.camel.karaf/apache-camel/2.8.5/xml/features
The same rule is applied to the ActiveMQ and CXF dedicated for every release of the ServiceMix.

This is pretty cool approach since it guarantees you that sticking to the given version of Camel (ActiveMQ, CXF, ...) will result in a stable ServiceMix installation. ServiceMix team invested pretty lots of effort to make sure that Camel in version X works nicely on ServiceMix in version Y. Fuse ESB is tested even more, to guarantee the highest level of the product.


Why stable dependencies are NOT cool

The downside of such stable versioning strategy is that you need to wait a long time before updating to the latest version of Camel (or CXF or whatever).

For example, if I committed Spring Batch component to the Camel and it has been released in Camel 2.10, then I'll need to wait for ServiceMix to use the higher version of Camel and Karaf. In this particular example Karaf 2.2.4 included in the latest stable version of ServiceMix (i.e. 4.4.2) doesn't support Camel 2.10.
karaf@root> features:install camel-spring-batch/2.10.0
Error executing command: Can not resolve feature:
Unsatisfied requirement(s):
---------------------------
   package:(&(package=jline.console.completer)(version>=2.7.0)(!(version>=3.0.0)))
      camel-karaf-commands

Bleeding edge ServiceMix

What to do if you really need to work with the latest version of the ServiceMix stack? Download the latest version of Apache Karaf and manually add Camel (CXF, ActiveMQ, etc) support into it.

For example to use the latest Camel (i.e. 2.10) I can download the latest Karaf (i.e. 2.2.9) and arm it with the Camel 2.10 features repository.
karaf@root> features:addUrl mvn:org.apache.camel.karaf/apache-camel/2.10.0/xml/features
Then I enable feature responsible for bootstrapping Camel modules backed by Spring.
karaf@root> features:install camel-spring
At last but not least I enjoy the brand new Camel modules.
karaf@root> features:install camel-spring-batch/2.10.0
Now I can drop my integration solution using the latest Camel 2.10 into my hand-made bleeding edge ServiceMix. :)

Tuesday, May 24, 2011

Apache Camel Fundamentals exam resurrected!

A long time ago I created a draft of the objectives for the Apache Camel Fundamentals exam. I want to reanimate the topic. The objectives have been reviewed and commented by the mighty Claus Ibsen himself so the quality of the recent version of Google Doc should be quite good. :)

I want to release the exam in the next week. Any feedback regarding the objectives is welcome and appreciated.

In general we (particularly I) plan to create separated track for ServiceMix stack. So expect more exams regarding Camel, ServiceMix, ActiveMQ, CXF and Fuse Source products in general. Here at Black Belt factory we just love ServiceMix and Fuse Source stuff, so we want to promote it and help people to learn this wonderful integration stack with our courses and exams.

If you are an integration kind of geek, you may also be interested in our new EIP exam.