Spring Integration - Camellos Continued

We had a great DevNexus conference this year and we were extremely blessed, that the project leads of both, Apache Camel (Claus Ibsen) and Spring Integration (Mark Fisher), spoke at the conference. In my opinion both frameworks are probably the best integration frameworks on the JVM!

Back in late 2009 I had written a blog post, where I presented some results from my Apache Camel learning experience. Now in 2011 it is time to do the same for Spring Integration. I am in the process of creating a few examples for Spring Integration that I create while learning it. So my hope is to create a little series of blog posts covering Spring Integration and Enterprise Integration Patterns (EIP) in general.

Nevertheless, don't forget to check out the wonderful examples that are provided via the Spring Integration Git repo at: http://git.springsource.org/spring-integration/samples/. It contains a sophisticated list of examples and is the best place to get started.

Well, let's get started - The source code for my example is available via GitHub, and will be successively expanded with additional examples in the coming weeks:

• https://github.com/ghillert/camellos-spring-integratio
• git://github.com/ghillert/camellos-spring-integration.git 

For right now the implemented core Spring Integration functionality in Camellos si is the following:

 

1. A user drops a file in a local directory (/camellos-si/data/inbox/), that is continuously polled by Spring Integration.
2. The file is picked up by Spring Integration and converted to a byte array with the original file being deleted.
3. The resulting byte array is sent to an in-memory channel.
4. A custom transformer will pick up the data array and compress the data using a ZipOutputStream
5. The resulting byte array is yet again being sent to another channel.
6. Finally an Outbound Channel Adapter will grab the data from the channel and send it via Ftp to an embedded Ftp server (stored under: /camellos-si/data/ftp/).

In order to try it out - just download camellos-si-distribution-1.0.zip from the downloads section at https://github.com/ghillert/camellos-spring-integration. Unzip it and start the application by executing:

java -jar camellos-si.jar

That's it. Spring Integration should boot up and you can drop files you want to zip up into '/camellos-si/data/inbox/'.

* * *

Some interesting additional features covered by my example:


Maven Shade Plugin (http://maven.apache.org/plugins/maven-shade-plugin/)

The Maven Shade Plugin can be a nice simplifier in stand-alone deployment scenarios. Let's say you have a project with multiple jar dependencies, then you can use this plugin to merge all jar dependencies into one single jar. You can also configure the Maven Shade Plugin so that it generates the Main-Class property in META-INF/MANIFEST.MF and consequently, you can deliver a single jar (No class-path hell) and execute the application like my simple example:

java -jar camellos-si.jar

Maven Assembly Plugin (http://maven.apache.org/plugins/maven-assembly-plugin/)

The Maven Assembly plugin helps with creating distributable packages. For instance for my example I need not only the compiled jar file but I also need a few directories in the right place, e.g. Apache FtpServer requires the users.properties file to be an actual file (having it as a class path resource embedded in your jar won't work).

Basically, you can use the Maven Assembly Plugin to move things around and to mold your resulting distributable package (a Zip file in my case).

Embedded Ftp Server using Apache FtpServer (http://mina.apache.org/ftpserver/)

Something I always find intriguing with Spring is the simple embedability of additional services such as running ActiveMQ embedded in your Spring context by just adding a single line of Xml, or in this case embedding Apache FtpServer. I know it is a blurry line where you start building your own application server but mentally I still find it easier to do it in Spring than using an app server, particularly when you only want to do it for testing, showcasing etc. and move on to use external ActiveMQ, Ftp Server instances for production etc.

I hope this example has been enlightening for your own Spring Integration discoveries - stay tuned for more. 

Apache Camel Alternatives

In part one of my mini series I provided a quick introduction to Apache Camel and in part two, I continued with a simple example illustrating the very basics to get you started (which I extended a bit in part three)

In this posting, I would like to talk about some of the alternatives that I see might be interesting for you to investigate, when considering an open-source enterprise integration framework. Hereby however, things always get a bit fuzzy in my opinion, depending on your business requirements.

On one side, you want to have components which allow you to communicate with other services, but on the other side you also want to have some sort of workflow-capabilities in order to orchestrate your business processes. Furthermore, you may be interested in monitoring and reporting functionality.  From a different view-angle, you may desire a higher-level solution versus a solution that embeds easily into existing code using plain Java APIs. The list could probably be expanded considerably, but I hope you get the picture.

Ergo, depending on your specific business requirements, the following list of frameworks may or may not provide a better choice compared to Apache Camel:    

• Spring Integration
• Spring Batch
  
• JBPM
• nexusBPM
  
• Drools
  
• Hadoop/Cascading
• Full ESBs such as Service Mix, Mule ESB, OpenESB, JBossESB

Actually, in my opinion the closest alternative to Apache Camel is Spring Integration.

Spring Integration

Before I was even aware of Apache Camel, I learned about Spring Integration at the No Fluff Just Stuff (NFJS) Java conference one year ago. Back then I thought that Spring Integration was the best thing since Nutella. Then a little later I learned about Apache Camel and tried to figure which one to take for a spin. As it turns out, comparative information on Google is rather sparse:

• Matt Raible has some interesting tidbits in one of his blog entries 
• There are some comments by Mark Fischer on the Spring Blog

Interestingly, Apache Camel provides a component that allows you to integrate Apache Camel with Spring Integration.

To me a good argument for Apache Camel was the greater availability of components compared to Spring Integration, as well as its adoption by other frameworks such as ActiveMQ and ServiceMix. 


Spring Batch

If your business requirement is to process large amounts of data, you may also want to look at Spring Batch which is specifically designed for batch processing requirements. There are some pointers on the Spring Batch website for integrating it with Spring Integration. I considered it for one of my past projects, but unfortunately I do not have massively deep experience with it as we ultimately chose JBoss jBPM.

jBPM

In my opinion another option worth considering is jBPM from JBoss. I used it in one of my past projects. Integrating it back then with Spring was certainly a  bit tricky (especially asynchronous execution of Nodes), but it is doable. Interestingly, starting with version 4.0, JBPM provides native Spring integration. Unfortunately, I did not have any opportunity, yet, to try out JBPM 4.0.  JBPM is certainly much heavier compared to Apache Camel or Spring Integration. I would say that its focus is also slightly a different one. While it provides support for human task management during workflow executions, managing the state of your workflow nodes etc., it totally looses out compared to the available components, which are provided by Apache Camel.


nexusBPM

A little bit jBPM on steroids is nexusBPM (An older blog post of mine about it is here). Developed by Intercontinental Hotels Group here in Atlanta, it is open-sourced at Sourceforge and provides a set of useful components in addition to the original JBpm distribution.

Drools

You probably would not necessarily think of Drools being a Workflow solution but what originally started out as a rules engine seems to venture increasingly into other areas as well. In fact, at least from a 30k foot view, it pretty much provides similar functionality as JBPM (Workflow, human task management etc.) and also provides a few components such as FTP, mail, executing system commands, etc.

What kind of baffles me is the fact that JBoss is maintaining 2 very similar products...

Cascading and Hadoop

If you need to process large amounts of file-based data, then Cascading might also be an interesting alternative. Cascading uses MapReduce to handle chunks of data. Interestingly, you can also use it for creating workflows. Not too long ago I attended an interesting presentation on that subject at AJUG. The presentation can be found here.

ESB like ServiceMix, Mule, OpenESB

Last in my list of alternative frameworks to Apache Camel is an entire category of solutions: Enterprise Service Buses (ESB). Enterprise Service Buses are basically stand-alone products, and as such, have a much heavier foot-print than for example Apache Camel which is basically just a set of light-weight Java APIs. In fact Apache Service Mix 4.0 is using Apache Camel under the hood in order to provide message routing. Besides Apache ServiceMix, you may also look into these OSS products:

• Mule ESB 
• OpenESB  
• JBossESB
• Apache Synapse (See: Comparison ServiceMix and Synapse)
  

This concludes my VERY high-level overview of frameworks/products that allow you to implement worklow and/or enterprise integration-based solutions. This whole subject could easily be extended into a more comprehensive article. Nevertheless, I hope I was able (within the limits of a blog post) to provide you with some good starting points.

¡Hasta la próxima!

Camellos III - Zipping Files with Apache Camel Example

I played a bit more with Apache Camel and I extended my little Camellos project (See this blogpost for details, including for directions to get to the source code) by another simple Example:

1. poll directory zip-input for any type of files
   
2. pack the files using the Zip data format
   
3. output files to the zip-output directory

I thought this would be an easy example to get my feet a bit more wet with Camel. Apache Camel provides 2 data formats for zipping files:

• GZip Data Format (Source Code)
• Zip Data Format (Source Code)

Interestingly, using the marshaller for the Zip data format, you're unable to generate standard zip file. Looking through the source code of the Zip daa format, it turns out that Apache Camel is using a DeflaterOutputStream to created the compressed data versus using a ZipOutputStream. A bug??

Thus, what a great opportunity to create my fist customized Camel Data Format implementation :-) I basically just took the existing ZipDataFormat class and converted it to using a ZipOutputstream, and by adding a single ZipEntry for my file. This works quite well.

I also added a reverse route for unzipping a file. One limitation exists. The zipfile can contain only one file. I am not sure, yet, how to solve the issue when you have more than one file in your zip file...I guess this may be a task for the Splitter or Aggregator components.

Camellos - Discovering Apache Camel II

As indicated in my last blog post, here is an example implementing a small Apache Camel example.

You can pick up the source code from:

• Checkout: http://jrecruiter.googlecode.com/svn/trunk/camellos/
• View: http://code.google.com/p/jrecruiter/source/browse/#svn/trunk/camellos

Steps to get it running:

1. Check out the source
2. Using Maven run: mvn camel:run
3. The Application should compile and start up correctly.
4. You can now drop files into the camellos/inbox directory
5. The files should get uploaded to its the FTP server running at localhost:3333
6. The uploaded files should show up under camellos/ftp

Back to y example, for my little blog post example here I want to provide the following very simplistic functionality:

1. pick up files from an directory
2. make sure that you pick up no more than 3 files per 30 seconds
   
3. store them into a JMS queue
4. have a listener on that queue that picks up those files
5. and upload files to a remote FTP site

What do you think, how many lines of Java code does it take?

With Apache Camel you can get this simple task done with ZERO lines of Java code. Well, I needed 1 Main class with a few lines of code to load the Spring context and and the embedded FTP server. Nevertheless, I think that is quite impressive. In a sense all the heavy lifting is done in the Spring Application Context file:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:camel="http://camel.apache.org/schema/spring"
xmlns:ftpserver="http://mina.apache.org/ftpserver/spring/v1"
xmlns:amq="http://activemq.apache.org/schema/core"
xsi:schemaLocation="
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd
http://mina.apache.org/ftpserver/spring/v1 http://mina.apache.org/ftpserver/ftpserver-1.0.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">


<amq:broker useJmx="false" persistent="false">
<amq:transportConnectors>
<amq:transportConnector uri="tcp://localhost:0" />
</amq:transportConnectors>
</amq:broker>

<bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
<property name="brokerURL" value="vm://localhost"/>
</bean>

<ftpserver:server id="ftpServer" max-logins="10"
anon-enabled="true" max-anon-logins="5" max-login-failures="3"
login-failure-delay="20">
<ftpserver:listeners>
<ftpserver:nio-listener name="default" port="3333" local-address="localhost"/>
</ftpserver:listeners>
<ftpserver:file-user-manager file="users.properties" encrypt-passwords="clear" />
</ftpserver:server>

<camel:camelContext shouldStartContext="true" trace="true">
<camel:package>com.hillert.camellos</camel:package>
<camel:route id="route1">
<camel:from uri="file:camellos/inbox?move=.done" />
<camel:throttle maximumRequestsPerPeriod="1" timePeriodMillis="10000" >
<camel:to uri="activemq:queue:camellos"/>
</camel:throttle>
</camel:route>
<camel:route id="route2">
<camel:from uri="activemq:queue:camellos" />
<camel:to uri="ftp://admin@localhost:3333?password=secret"/>\
</camel:route>
</camel:camelContext>
</beans>


Apache Camel provides its own Maven plugin: http://camel.apache.org/camel-maven-plugin.html

Getting started with Apache Camel is simple. I recommend using m2Eclipse which is a Maven plugin for Eclipse. Basically Camel provides it's own Maven archetype which after running creates a simple project structure and which can be immediately run using mvn camel:run after project creation.


For my implementation I followed Camel's tutorial for creating a Spring based Camel route.
As my example uses a few more components for Apache Camel but also ActiveMQ for JMS and Apache Mina FTP Server, I needed some additional Maven dependencies.

Getting all the Maven dependencies right took actually longer than implementing the actual application logic. Anyway, I hope this gives you a quick overview of some basic Apache Camel features. As time permits I will blog about more about it soon. See you then!

PS: Apache Mina FtpServer, is itself a nice little nifty package. So if you for example have the need to boot-up dynamically FTP servers from within your application...check it out.

Camellos - Discovering Apache Camel I

Over the next couple of weeks or months (depending how much spare-time I am able to allocate), I will dive into the world of Apache Camel (Also take a look at my second blog post)


Apache Camel is somewhat like a Swiss-army knife. As an integration framework (Message Routing API, Mediation Router), it implements all the Enterprise Integration Patterns from the book with the same name.


Next, Apache Camel provides a quite extensive component library supporting an impressive amount of communication protocols. Also, it support an wide range of data formats as well as integration points in terms of other frameworks such as Spring, Guice, ServiceMix, ActiveMQ et cetera.

Well, that's all nice and dandy—However, what does this mean for me as a software developer?

When you develop typical Java enterprise applications, you will sooner or later come across the requirement to connect to other systems or to add other asynchronous services to your applications. Something like:

"Hey, we need to pick up this file that comes in every night, parse it, process it and then stuff its data into our database and, while we're at it, also make sure the original file is archived somewhere in the file system."

Or maybe you just have some requirements where your application needs to send data to another server but you have to make sure, that the flow of data is "throttled" in order not to overburden your destination server during peak-processing times.

In all those cases Apache Camel can greatly simplify the implementation effort. It is basically a mini ESB in the form of a simple Java API. But it is extremely modular so you can just bits and pieces from it. In my next blog post I will show you an example showing you how it is really dead simple to get started with Apache Camel.

See second Camel related blog post.