Thursday, May 24, 2012

Aggregating multiple web services behind Camel facade

I've found an interesting problem on Stack Overflow. I've answered the question but want to elaborate it a little bit.

The problem

You got two web services. Service A and Service B. The services are independent from each others. You need to create a REST web service that calls the latter services, aggregates the results and returns them to the client.

Some terminology

What we want to achieve here is an example of simple orchestration i.e. coordination of calling of multiple services on the ESB. Orchestration of the services (not necessarily web ones) has been repeatedly raped by many commercial tools and solutions. One of the most prominent example of orchestration's torturer is BPEL.

When we use ESB as a proxy between services we refer to it as Pure ESB. Pure ESB doesn't contain heavyweight processing logic - it only serves as a mediation router.

ServiceMix and Camel are excellent tools for creating lightweight orchestration based on the pure ESB concept.

Aggregation of two or more services can be referred as Aggregating Service, Facade Service or Proxy Service.


Aggregating web services in Camel

We want to call Service A and Service B in parallel (since they are independent), then aggregate the incoming results into a single answer.

The following Camel route is responsible for collecting results from the sub-services (A and B) and aggregating them into single message.

from("direct:serviceAggregator")
  .multicast(new GroupedExchangeAggregationStrategy()).parallelProcessing()
    .enrich("http://servicea.com").enrich("http://serviceb.com")
  .end();


We use multicast pattern to concurrently call two web services. Then multicast uses message enricher pattern to add data from the external services to the original message. When Camel finally finishes aggregation of the results of A and B services' calls, it attaches the latter results to the original message passed to the direct:serviceFacade endpoint. The messages fetched from the A and B services will be stored as exchange property identified by Exchange.GROUPED_EXCHANGE key.

To expose our facade service F on the ESB we will use Camel's Jetty component.

from("jetty:http://0.0.0.0:8080/myapp/myComplexService").enrich("direct:serviceFacade").setBody(property(Exchange.GROUPED_EXCHANGE));

We use enricher again - this time each message passed to the Jetty HTTP connector will be enriched by our aggregating route (direct:serviceAggregator). In practice that means that all request to the HTTP connector exposed as our service will be injected with two additional messages from A and B services. In this particular example we extract data from the received messages and render them as HTTP response to the client.

7 comments:

  1. I have a simple question, I am new to Apache Camel and still trying to understand how to use Camel for integrating two webservices or such a scenario.

    Lets say I have two Systems to integrate(SystemA and SystemB) and of course some adaptation is required as the two corresponding endpoints are not one to one mapping to each other. Lets name those end points as WebServiceA(SOAP based) and WebServiceB(REST based).

    As an unfamiliar guy with ESB/Camel technologies, the first thing come to mind is that to create another webservice to meditate WebServiceA and WebServiceB. Lets name it as WebServiceC , so the business logic to act as an adatper and invoking the WebServiceB is a responsibility of WebServiceC. So I didn't use Camel here. So the SystemA will invoke the WebServiceC and it will do the adaptation and invoke WebServiceB which will integrate SystemA and the SystemB.


    1. Could you please explain me how I can improve my integration by using Apache Camel here ???

    2. The approach I can see here on how to squeeze Camel in is, Create another WebService (WebServiceC) and create a message channel from that webservice to WebServiceB(Consumer) . And message translation to be done using some POJOs adhering with Process and Bean concepts. . Appreciate if you could explain this.

    ReplyDelete
  2. Hi Prasanna,

    The main advantage of using Camel here that you can avoid creation of aggregating service. The Camel DSL code demonstrated in the blog post is all you need to orchestrate the aggregation. In case of bigger services and more complex orchestration, this is a real advantage that would save you typing a lots of boilerplate code (and debugging of the latter).

    Value added by Camel here is also it's ability to handle messaging-related issues (like monitoring, redelivery control, reliable failures handling and so forth).

    I hope I put some light on your doubts here :) .

    ReplyDelete
  3. Can you help provide the above code in a camelContext or blueprint xml? I'm trying to configure the above in a xml file but just struggling with it.

    ReplyDelete
  4. Hi Rahul,

    Here is the XML DSL version for you:

    ...
    <route>
    <from uri="direct:test"/>
    <multicast parallelProcessing="true" strategyRef="group">
    <pollEnrich uri="seda:a"/>
    <pollEnrich uri="seda:b"/>
    </multicast>
    <to uri="mock:test"/>
    </route>

    ...

    <bean id="group" class="org.apache.camel.processor.aggregate.GroupedExchangeAggregationStrategy"/>

    Best regards,
    Henryk

    ReplyDelete
  5. I have somewhat similar requirement but the producer services are dependent one ! My requirement is as below
    There are 2 REST web service offered by 2 different producer application. service1 will provide response containing order details & item ids associated with the order number. Service2 will provide item details for the given item IDs (in json data structure). The consumer application will make a call to Camel with order number. Camel has to make call to Service1, get order details plus the item IDs associated with Order. then, Camel has to extract the item ID's from first service response & pass Item IDs as parameter to Service2 to get Item details. Later merge the order details from Service1 & item details of Service2 (remove Item Id's & replace with Item Details) & respond to consumer application (as one JSON structure).

    Could you suggest suitable camel solution please ?

    Regards,
    Ravi

    ReplyDelete
  6. Hi Ravi,

    If you just need to call two services in a sequence (passing input from the first one to the second one) you can use enricher without multicast.

    from("direct:start").
    enrich("http://service1.com").
    .setHeader("valueFromService1", body())
    enrich("http://service2.com").
    to("mock:agregated");

    Cheers.

    ReplyDelete
  7. Hi. can you provide a complete example with the enrich implementations. im unable to make it works. please in sring DSL

    ReplyDelete