Saturday, September 26, 2009

Proxying large Amounts of Data using UrlRewriteFilter

Finally, I found some time to submit a feature request with patch to the UrlRewriteFilter project.

http://code.google.com/p/urlrewritefilter/issues/detail?id=53

In one of my projects we needed to provide functionality to post (upload) and download data via a servlet container to/from another url/port - basically we needed to implement proxying.

A great library out there is UrlRewriteFilter, a Java library that provides Apache's mod_rewrite functionality. Not only can you use it to make complex urls more user-fiendly, or re-map old url to new ones but it also provides proxying capabilities.

UrlRewriteFilter uses Apache HttpClient for doing proxying. Unfortunately, I ran into memory issues when proxying large amounts of data.  The issue is that the current version of UrlRewriteFilter (3.2) is doing buffered requests while proxying. This probably works fine for 90% of all use-cases but for the project I am working on we need to basically support unlimited amounts of data to be proxied (multiple 100s of MB).

Thus I provided a patch, that worked really well in my project without increasing memory consumption.
In Apache HttpClient you can implement a custom class using the RequestEntity interface that allows you to stream the data directly.

Thursday, September 24, 2009

Camellos - Discovering Apache Camel II

As indicated in my last blog post, here is an example implementing a small Apache Camel example.

You can pick up the source code from:
Steps to get it running:
  1. Check out the source
  2. Using Maven run: mvn camel:run
  3. The Application should compile and start up correctly.
  4. You can now drop files into the camellos/inbox directory
  5. The files should get uploaded to its the FTP server running at localhost:3333
  6. The uploaded files should show up under camellos/ftp
Back to y example, for my little blog post example here I want to provide the following very simplistic functionality:
  1. pick up files from an directory
  2. make sure that you pick up no more than 3 files per 30 seconds
  3. store them into a JMS queue
  4. have a listener on that queue that picks up those files
  5. and upload files to a remote FTP site
What do you think, how many lines of Java code does it take?

With Apache Camel you can get this simple task done with ZERO lines of Java code. Well, I needed 1 Main class with a few lines of code to load the Spring context and and the embedded FTP server. Nevertheless, I think that is quite impressive. In a sense all the heavy lifting is done in the Spring Application Context file:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:camel="http://camel.apache.org/schema/spring"
xmlns:ftpserver="http://mina.apache.org/ftpserver/spring/v1"
xmlns:amq="http://activemq.apache.org/schema/core"
xsi:schemaLocation="
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd
http://mina.apache.org/ftpserver/spring/v1 http://mina.apache.org/ftpserver/ftpserver-1.0.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">


<amq:broker useJmx="false" persistent="false">
<amq:transportConnectors>
<amq:transportConnector uri="tcp://localhost:0" />
</amq:transportConnectors>
</amq:broker>

<bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
<property name="brokerURL" value="vm://localhost"/>
</bean>

<ftpserver:server id="ftpServer" max-logins="10"
anon-enabled="true" max-anon-logins="5" max-login-failures="3"
login-failure-delay="20">
<ftpserver:listeners>
<ftpserver:nio-listener name="default" port="3333" local-address="localhost"/>
</ftpserver:listeners>
<ftpserver:file-user-manager file="users.properties" encrypt-passwords="clear" />
</ftpserver:server>

<camel:camelContext shouldStartContext="true" trace="true">
<camel:package>com.hillert.camellos</camel:package>
<camel:route id="route1">
<camel:from uri="file:camellos/inbox?move=.done" />
<camel:throttle maximumRequestsPerPeriod="1" timePeriodMillis="10000" >
<camel:to uri="activemq:queue:camellos"/>
</camel:throttle>
</camel:route>
<camel:route id="route2">
<camel:from uri="activemq:queue:camellos" />
<camel:to uri="ftp://admin@localhost:3333?password=secret"/>\
</camel:route>
</camel:camelContext>
</beans>


Apache Camel provides its own Maven plugin: http://camel.apache.org/camel-maven-plugin.html

Getting started with Apache Camel is simple. I recommend using m2Eclipse which is a Maven plugin for Eclipse. Basically Camel provides it's own Maven archetype which after running creates a simple project structure and which can be immediately run using mvn camel:run after project creation.


For my implementation I followed Camel's tutorial for creating a Spring based Camel route.
As my example uses a few more components for Apache Camel but also ActiveMQ for JMS and Apache Mina FTP Server, I needed some additional Maven dependencies.

Getting all the Maven dependencies right took actually longer than implementing the actual application logic. Anyway, I hope this gives you a quick overview of some basic Apache Camel features. As time permits I will blog about more about it soon. See you then!

PS: Apache Mina FtpServer, is itself a nice little nifty package. So if you for example have the need to boot-up dynamically FTP servers from within your application...check it out.

Wednesday, September 23, 2009

Camellos - Discovering Apache Camel I

Over the next couple of weeks or months (depending how much spare-time I am able to allocate), I will dive into the world of Apache Camel (Also take a look at my second blog post)


Apache Camel is somewhat like a Swiss-army knife. As an integration framework (Message Routing API, Mediation Router), it implements all the Enterprise Integration Patterns from the book with the same name.


Next, Apache Camel provides a quite extensive component library supporting an impressive amount of communication protocols. Also, it support an wide range of data formats as well as integration points in terms of other frameworks such as Spring, Guice, ServiceMix, ActiveMQ et cetera.

Well, that's all nice and dandy—However, what does this mean for me as a software developer?

When you develop typical Java enterprise applications, you will sooner or later come across the requirement to connect to other systems or to add other asynchronous services to your applications. Something like:

"Hey, we need to pick up this file that comes in every night, parse it, process it and then stuff its data into our database and, while we're at it, also make sure the original file is archived somewhere in the file system."

Or maybe you just have some requirements where your application needs to send data to another server but you have to make sure, that the flow of data is "throttled" in order not to overburden your destination server during peak-processing times.

In all those cases Apache Camel can greatly simplify the implementation effort. It is basically a mini ESB in the form of a simple Java API. But it is extremely modular so you can just bits and pieces from it. In my next blog post I will show you an example showing you how it is really dead simple to get started with Apache Camel.

See second Camel related blog post.

Saturday, September 19, 2009

GWT - Hosted Mode Gotcha in Windows

I ran into a small gotcha while running a GWT application I am working on in hosted mode. Usually I am developing my application in hosted mode but for quick stand-alone deployments I also need to deploy the application to a dedicated servlet-container. Since the default settings cause the application to compile rather slowly, I have been trying to speed up GWT compilation times (Using GWT 1.7) by compiling against Firefox only (My browser of choice).

To achieve this I set the following property in my GWT module's *.gwt.xml file:



Well, as it turns out, this caused me quite some pain in hosted mode. I did not realize that the aforementioned setting affects hosted mode. I new that under Windows Internet Explorer (IE) is the default browser but always assumed it only affects fully compiled GWT code - not code running in hosted mode. That was a painful lesson - It caused some rather obscure errors in hosted mode and nowhere were I able to find explicit information regarding this issue.

Sunday, August 30, 2009

jRecruiter is deployed and live

Finally...it took ways to long. Last Sunday I finally made a new production deployment (Version 2.0) of jRecruiter). jRecruiter is the job posting service of the Atlanta Java Users Group (AJUG). This version was a long time in the making. I hope that from now on I can make new feature deployments more swiftly. The new version is accessible at: http://www.ajug.org/jrecruiter/

Here is a list of changes:
  • (Almost) No more Struts XML-configuration files. jRecruiter uses the Struts 2.1 convention plugin for annotation based configuration.
  • Many Spring Beans dependencies are now configured using annotations (Spring 3.0)
  • Sophisticated data-grid support using jMesa
  • Google Maps integration (web (Javascript), Pdf (Image) and Flex (Flash))
  • Hibernate Search support
  • RSS feeds for the latest 20 jobs using ROME
  • Social bookmarks on the job posting list
  • Application is now Struts 2.1 based
  • Using jasypt for password digesting
  • Administrative pages are SSL encrypted
  • Improved charting (JFreeCharts)
  • Using reCAPTCHA for the registration page
  • Implemented email account verification
  • Now job postings and updates are send to Twitter (http://twitter.com/ajug_jobs/)
  • PDF Export of Job Posting Details (using iText)
  • Switched from username-based logins to an email-based system (usernames are still supported on the backend (e.g. administrative logins)
  • Some performance improvements --> CSS and JavaScript files are now merged, minified and GZipped using JAWR)
  • XML web services for providing job posting data to Indeed.com (Still need to do the actual integration but the XML feed is there)
  • Using jQuery for most Javascript needs
  • Maven project consists of multiple modules now (web, server, flex)
  • I18N: Added partial support for German
  • Extra bonus: Added a simple Adobe Flex front-end using Spring Blaze DS for server integration

Wednesday, July 29, 2009

Integrate Twitter and bit.ly into your Java App

I am in the final stage of developing towards the 2.0 release of jRecruiter. It is the job posting service of the Atlanta Java Users Group (AJUG) and as such I devote some of my spare-time towards it. It is a great way to explore all kinds of different technologies. Thus, last weekend I had the idea that it might be a cool idea to let the system "tweet" about job posting additions and updates. Here are my findings.

First I looked at available Twitter libraries for Java:
After some considerations, I chose Twitter4J. To me, it seemed to be the most active of the three projects. Even better, it has a BSD-style license. Using Twitter4J is super simple:

Twitter twitter = new Twitter("username", "password");

try {
twitter.updateStatus("My tweet");
} catch (TwitterException e) {
…
}

There is so much more you can with Twitter4J. Check out its documentation for further details.

Secondly, as each tweet contains a link back to the job detail page in the main application, I was looking into ways to shorten URLs. That way, I can store more job posting related data within the only 140 characters available for each Twitter status update.

The original service for this is TinyURL. However, it looks like bit.ly has more traction these days and is also used directly by Twitter itself. The nice thing about bit.ly is that they have developer accounts and are offering a web service API for doing URL shortening.

Even better, there is already a Java library available for using bit.ly called bitlyj. Using bitlyj is actually almost as equally simple to use as Twitter4j:

final Bitly bitly =
BitlyFactory.newInstance("username", "your_bitly_api_key");
try {
BitlyUrl bUrl = bitly.shorten("http://www.google.com/");
URL url = bUrl.getShortUrl();
} catch (IOException e) {
…
}

Unfortunately, you have to build bitlyj yourself, but thanks to Maven it is a piece of cake. One thing I noticed, is that the error handling of the library could be more robust. Accidentally, I was using my bit.ly password, initially. However, bit.ly provides an API Key. By the way if you ever wonder where you can get that API Key-It is right in your account page. Anyway, the library needs some attention but for my simple use-case it worked fine.

Thursday, July 23, 2009

MapReduce, Hadoop and Cascading at AJUG

Yesderday Chris Curtin, CTO at Silverpop, gave an awesome presentation on MapReduce, Hadoop and Cascading. I really enjoyed his wealth of real-world practical examples.

His presentation is available at: http://www.slideshare.net/chriscurtin
Thanks to John Willis, there is also a video recording available at: http://www.johnmwillis.com/ec2/hadoop-and-cascading-ajug-072109/

Sunday, June 7, 2009

Book Review: Spring Web Flow 2 Web Development

Spring Web Flow 2 Web Development by Sven Lüppken and Markus Stäuble is the currently only available book on Spring Web Flow 2.0. The book covers all the components that are part of the Spring Web Flow distribution such as:
  • Spring Web Flow
  • Spring Faces
  • Spring JavaScript
Additionally, the book also provides an overview of Spring Security. The book comes with example code for many of the chapters.

The provided examples, however, could have been a little more consistent. The main source example in this book is covered in chapter 5 (A simple bug tracker). I liked the given example, as it does not require a database. This makes it easy to understand the Spring Web Flow specific problem domain. However, considering the fact that the authors also talk about Spring Security in other chapters, it would have been nice, if they had used the bug tracker example throughout the book in order to create a “complete” application, that ties together Spring Web Flow, the persistence layer and Spring Security. Unfortunately, the code examples given in later chapters don't refer back to the bug tracker example. Instead, chapter 6, for example, uses a shopping cart example.

A further minor annoyance is that the source code for 2 of the chapters still contain the eclipse project files used by the authors. This may be confusing for some users as these project files still point to Microsoft SQL Server drivers on a windows D: drive. I, for instance, use a Mac...

In chapter 2 the book discusses on 4 pages Ant and Maven. I think it might have been simpler for the understanding of the book and the provided source code, if only one build system was used, e.g. Maven. Some of the code examples provide Ant and Maven build scripts but some other ones provide support for Maven, only.

Another anomaly was, that the book describes the installation of the Spring IDE plugins for Netbeans and Eclipse on 7 pages but then the authors mention in passing only that they are using Microsoft SQL Server as underlying persistence store for their example. As many Java developers may not have a MS SQL Server license, it may have been better to use open-source Databases such as MySQL or PostgreSQL.

The example in Chapter 3 uses HSQLDB for its database but earlier (Chapter 2) the authors use MS SQL Server. For consistency and simplicity reasons the book could have stuck with one database.

Chapter 4 deals with Spring Faces. The authors should have mentioned JBoss Seam, since Seam solves a similar problem as Spring Web Flow but having a default focus on JSF.

The book provides an entire chapter on JSF integration (Chapter 4). As JSF is a widely used framework and Spring Weblow provides native support for JSF. It would have been nice though, if support for other frameworks such as Struts 1 and 2 would have been illuminated.

As part of chapter 5, the book also provides information on Spring JavaScript. While it is part of the official Spring Web Flow distribution it is not necessarily essential for the working of Spring Web Flow and even its spec lead Keith Donald mentioned at the Atlanta Java Users Group (AJUG) meeting in November'08 that it maybe extracted into its own project in the future.

The section in chapter 5 which starts on page 165, the Web Flow Configuration, felt repetitive to parts of Chapter 3 (pages 53) and should have probably merged with that section. Chapter 5 also contains a larger section on Apache Tiles integration. One thing that I did not understand was how Tiles matters in terms of mastering Spring Web Flow. Generally I don't mind explaining additional pieces of information around the edges of your problem domain but if that happens on 5 percent of the book, it needs to matter within the context of the targeted subject. On a site-note: Take a look at SiteMesh for your templating needs. I strongly prefer it compared to Tiles.

In chapter 6 the testing of your Spring Web Flows is explained. This was an informative chapter and I particularly liked mentioning of EasyMock.

In chapter 7 the book primarily details Spring Security. In my opinion, it provides too much Spring Security centric information. For example, the chapter describes how to create and write your own AccessDecisionVoter. In my opinion it does not help and is unnecessary for the understanding of Spring Web Flow. At most I would have chosen an example that uses a DB-backed security implementation using default Spring Security functionality.

To me, chapter 7 was a bit strange. It provides 16 pages of information introducing Spring Security but the actual integration of Spring Web Flow with Spring Security is accomplished in merely 4 pages. Most interestingly, that's where the book simply ends. I missed something like a wrap-up, that summarizes what the book covered and where to continue on.

In Appendix B the authors provide an overview of Springsource's dm Server. Personally, I think that dm Server is an interesting offering. However, I don't understand why this appendix even exists in this book. It is neither important for the understanding of Spring Web Flow, nor do the authors explain why this appendix exists.

Here are some further thoughts: Throughout the book, I like the use of Maven for building the examples. It certainly makes it easy to understand the examples' structure.

I think for the understanding of Spring Web Flow, it unnecessarily complicates things, if you present examples that require JNDI for establishing database connectivity. Furthermore it binds you to a concrete application server, which in this case is Tomcat.

Except for a for a very brief mentioning on page 224, the book never talked about the applicability of Spring Web Flow in the context of RIA frameworks such as GWT, Flex etc. Is there a use case for Spring Web Flow e.g. for coarse grained server state for large, complex applications?

It would have been nice if the book provided an overview of other frameworks that handle conversational state such as JBoss Seam. Additionally, since the books was published in March 2009, it could have described some of the planned conversational state management support planned for Spring 3.0.

Furthermore, it would have been great if the book had also mentioned how Terracotta can be used to back Spring Web Flow conversations. Terracotta provides an interesting reference application detailing that possibility.

Anyway, the book follows generally the Spring Web Flow documentation. The question is how much more value does the book add compared to the reference documentation. Overall, I have a quite conflicted opinion about this book. I think, if you are interesting in JSF integration or the bulk of the additionally discussed technologies (E.g. Spring JavaScript), then this book may add enough information for you. It certainly provides some useful information regarding Spring Web Flow 2.0.

Unfortunately, it contains quite a few inconsistencies and it is overall not an easy read. However, it is the only currently available book dealing with Spring Webflow 2.0. On Amazon, I gave the book 2 out of 5 stars.

Wednesday, June 3, 2009

Book Review: Learning Flex 3

Learning Flex 3 by Alaric Cole is a great introductory book for Adobe Flex 3. If you have some basic experience with HTML, JavaScript and CSS, then you will learn the basics of Flex rapidly using this book as a guide. Some of the areas that are taught are:
However, if your professional background is in web-application development using Java or other OO languages, then you are probably more likely to skip or cross-read sections of the book. This is because the book starts out with practically no initial pre-requirements, and thus you might be familiar with some of the discussed areas.

The book gave me an excellent overview of Flex's capabilities, and they are all explained thoroughly and are easily understandable. Also, this may sound silly, but I particularly liked the refreshing fact that the book is in color, unlike most other IT books.

Furthermore, the book explained Data Binding (Chapter 7) very well. This is something I was not familiar with coming from the Java world. The chapter illustrates the concepts of one-way binding, two-way binding and making your own variables “bindable” using a metadata declaration, which is basically an annotation in Java parlance.

Another feature I liked a lot was View States (Chapter 12), which allows you to rearrange, group and reuse components within your application.

I wish the book provided more information for further reading. I understand that certain concepts are beyond the scope of this book, but it would have been nice if those had been mentioned and links or recommendations for further reading were provided.

For example, while AMF is mentioned on page 8 and on page 157 the book briefly talks about the Webservice and RemoteObject component, the book should have also mentioned BlazeDS as one of Flex's core technologies for communicating with back-end servers.

And for more complex applications, the book could have at least pointed out some of the available MVC frameworks for Flex and some pointers of where to read more about them (E.g. Cairngorm and PureMVC). Well, and then there is Degrafa, the declarative graphics framework…

While the author explains the aspects of using CSS in Flex applications very well, he could have further stressed that Flex uses a subset of CSS, which in certain areas behaves differently compared to CSS you more typically use in HTML pages. He should have enumerated some of those pitfalls.

Having said all this, these issues mentioned above are minor in nature. Overall, the book was a fun read! And particularly to Java web-developers, Flex may very well be THE contender for application user interfaces moving forward. One of the great things about Flex is that even the standard components look very good (and should be good enough to please your boss), and your application looks and behaves consistently across various browsers. Oh, and yes you can run the same application on the desktop as well (Chapter 15).

This book will definitely help you learn and master Flex, and you should be able to produce functional user interfaces quickly. In order to learn more about integrating Flex with your Spring powered back-end you may want to also consider looking at “Pro Flex on Spring”.

Thursday, May 14, 2009

JavaRebel + m2eclipse = Sweet Spot

One of the pain-points while developing Java-based web-applications is the issue of the bouncing server. Every time you change a class file, your web application context needs a restart. While this is mildly annoying for small applications, it is simply unbearable for anything more substantial.

As far as I now the best and cheapest solution so far has been to use MyEclipse. It worked great and I rarely had to restart web contexts.

I always wondered why the standard IDEs (E.g. plain Eclipse WTP, IntelliJ ) never solved that re-deployment issue. Well, at least I have not been able to get this sufficiently working in plain Eclipse+WTP or IDEA IntelliJ.

Unfortunately, with the advent of Maven, MyEclipse started to loose come of its charm. For years, they were not able to provide an acceptable Maven integration (There was an entry on their forum for years demanding that feature) and on the other side, WTP started to become usable (I have not really tested the Maven integration introduced with MyEclipse 7 but I have read, that it is still not perfect). But one thing, that MyEclipse had nonetheless going for it, were its hot deployment capabilities.

Enter ZeroTurnaround's JavaRebel. Four weeks ago I started looking at JavaRebel and have been using it for the development of my OSS project since then. My verdict: It is awesome.

I use it in conjunction with the m2eclipse plugin and it simply works. ZeroTurnaround provides guides regarding the Eclipse and Maven integration (here and here) that are fairly straigtforward. It also comes with support for Spring and Struts2 directly.

The nice thing about JavaRebel is that it is agnostic to the used IDE as it simply hooks into the JVM as a Java agent. Therefore, you probably could also easily use it from the command line (in case you like to code using VI :-).

One drawback is the price which makes JavaRebel more expensive than MyEclipse. On the other hand, there is not much in MyEclipse that you cannot have in plain Eclipse/WTP using free plugins. Thus, ultimately it may boil down to personal preference but to me JavaRebel with plain Eclipse WTP (or IntelliJ for that matter) is the much nicer package. And if you have a project with an OSS license you can get JavaRebel for free, and don't have to buy MyEclipse.