TomcatExpert

Basic Apache Tomcat clustering for Grails applications

posted by pledbrook on July 20, 2010 06:43 AM

Grails is a rapid application development framework for web applications on the Java platform. It's similar in many respects to Ruby on Rails, but it's based on Java libraries, like Spring and Hibernate, and the Groovy language. It also produces standard WAR files that can be deployed to a servlet container like Tomcat. That means you can deploy Grails applications to a Tomcat cluster and in this article I'll show you how.

Getting started

In order to demonstrate clustering, we need an application to deploy. Assuming that Grails is installed (I was using Grails 1.3.3 - the latest) we can create a brand new application with the command

grails create-app my-cluster-app

This will create a my-cluster-app directory containing the project. If we then switch to that directory, we can generate a WAR file straight away by running

grails war

Of course, the application doesn't do anything yet. Nor is it ready for clustering.

Those in the know are aware that a Java web application can't be clustered unless it is marked as 'distributable' and Grails applications are not by default marked as such. The easy solution is to add the <distributable/> element to the web descriptor, but you won't find a web.xml file in a standard Grails application. That's because Grails generates the web descriptor on the fly when it needs one. Fortunately, a web.xml that you can edit is only a command away:

grails install-templates

This will install several template files in your project, but the one we're interested in is src/templates/war/web.xml. Open this up and add the <distributable/> element:

<web-app ...>
  <display-name>/@grails.project.key@</display-name>
 
  <!-- Add this line -->
  <distributable/>
  &hellip;
</web-app>

Now when Grails generates the WAR file, the web application will be distributable.

It would be great if this was all we needed to do to make a Grails application ready for clustering, but that bane of deployments everywhere, configuration, must be tackled first. In a standard Grails application, the relevant files are packaged with the WAR file. This is convenient for development since you don't have to worry about managing external configuration files and their locations, but it prevents you from customising the configuration per deployment. What's the solution?

Local configuration

There's no getting away from the fact that the simplest way to provide per-deployment configuration is via external configuration files located on the deployment servers. This allows you to reconfigure the application at any time with a quick edit and server restart rather than a redeployment of the WAR. Grails allows for external configuration files through a special grails.config.locations setting in the grails-app/conf/Config.groovy file:

grails.config.locations = [ "file:./${appName}-config.groovy", "classpath:${appName}-config.groovy" ]

This tells Grails to load two configuration files, one from the local filesystem and one from the classpath. Why the reference to 'appName'? It's so that you can easily host multiple web applications on a server/cluster without the names of the configuration files conflicting with each other. At runtime, ${appName} is replaced by the name of the application, i.e. the name passed to the create-app command - 'my-cluster-app' in this case.

The first path is for development when you're executing grails run-app - the configuration file just goes in the root of the project. The second item is for WAR deployment and causes Grails to look for the file on the classpath. You often don't know at development time where the current working directory is or where configuration files go on the deployment server's file systemm. So where can you put the configuration file so that it goes on the classpath? Since Tomcat add its lib directory to the classpath, that's the ideal location. Don't worry if Grails can't find one or other of the specified files: it will simply print a warning to stderr but continue to start up normally.

We'll return to these configuration files and what to put in them later when we come to the actual deployment. That story isn't over yet. For now, I want to consider features that would be affected by clustering, such as caching, scheduling, etc. It's easy to overlook them because Grails typically makes it very easy to include features without explicit configuration, but if you deploy to a cluster with their default settings, they will typically break.

Distributed caching

When you first create a Grails application, it has Hibernate's 2nd-level cache already enabled and configured to use Ehcache. You can take a look at the grails-app/conf/DataSource.groovy file to confirm this - you should see the following:

hibernate {
    cache.use_second_level_cache = true
    cache.use_query_cache = true
    cache.provider_class = 'net.sf.ehcache.hibernate.EhCacheProvider'
}

As you can see, it configures Ehcache as the cache provider. What you won't see is an Ehcache configuration file (ehcache.xml) in the project. That means Ehcache will use its default settings. This isn't recommended for a standalone server, let alone a clustered one. An additional concern is that Ehcache should be configured differently for local development (using grails run-app) and deployment to the cluster. How do we manage that?

Ehcache will load the ehcache.xml file from the classpath. We can easily make sure that the file ends up on the application's classpath by placing it in either the grails-app/conf or src/java directories. The trouble is, that file will be used in both development and the deployed web application. If you want different files for development and deployment you have to resort to a clever trick: replacing the file at packaging time.

The Grails build system allows the developer to interact with the build via an event system. One of those events is triggered just before the WAR is created, which allows us to influence what goes into the final package. All we have to do is create a scripts/_Events.groovy file containing this code:

eventCreateWarStart = { name, stagingDir ->
    ant.copy(file: 'cluster_resources/ehcache_distributed.xml',
             tofile: "$stagingDir/WEB-INF/classes/ehcache.xml", overwrite: true)
}

This event handler will copy the distributed cache configuration file from the cluster_resources directory to the staging directory, replacing the standard ehcache.xml. Once its done, Grails zips up the staging directory as the WAR file. Job done. Note that cluster_resources is not a standard Grails directory, so it must be created manually. You can find it along with the Ehcache configuration files in the attached project.

Caching isn't the only case where you have to do something special for cluster deployment, but external configuration files and tinkering with the generated WAR file will typically work for other cases as well. For example, you can use the same technique we used for ehcache.xml to ensure that an appropriate quartz.properties file (if you're using Quartz) is used by the application on the cluster.

We're now almost ready for deployment, but we don't yet have a way to test the session replication. For that, we need some web pages that utilise the HTTP session.

Creating the test pages

This being Grails, creating test pages is straightforward. We're going to use a single controller with three actions (which correspond to three distinct pages): 'arrive', 'leave', and 'counter'. Create the file grails-app/controllers/HomeController.groovy and give it this content:

class HomeController {
    def arrive = {
        if (session.counter) {
            render "You are already here"
        }
        else {
            session.counter = 1
            render "You have arrived - counter initialised"
        }
    }
 
    def leave = {
        if (!session.counter) {
            render "You have already left"
        }
        else {
            session.counter = null
            render "You have left - counter cleared"
        }
    }
 
    def counter = {
        if (session.counter) {
            session.counter++
            render "Counter: ${session.counter}"
        }
        else {
            render "You're not here!"
        }
    }
}

When we hit the page /home/arrive the counter variable will be added to the session. This counter will then be incremented and displayed each time we go to the /home/counter page. Finally, the variable is removed from the session when we hit /home/leave. Simple, pointless, but it does demonstrate the session behaviour of the application. We can now run the grails war command to generate a WAR file.

Deployment

Now that we have a WAR file, we can focus on deploying it to a Tomcat cluster. It's not the focus of this article to detail how to set up a Tomcat cluster, but in case you haven't done it before I have attached a package of scripts based on Burt Beckwith's work and using Tomcat 6.0.28. Just download the tomcat-cluster-scripts.zip archive and unpack it to any directory, for example in ~/tomcat-cluster-scripts. In the root of the archive you will see several shell scripts which make creating and managing a Tomcat cluster almost trivial.

Let's start by creating a cluster in a /opt/java/tomcat-cluster directory:

cd ~/tomcat-cluster-scripts
export CR=/opt/java/tomcat-cluster
sudo ./createCluster.sh my-cluster-app

This will create a cluster for the application 'my-cluster-app'. You can pass the location of the cluster to all the scripts, but it's easier to create a CR environment variable containing the location and run the scripts without that argument. Saves typing! Our cluster is now ready and all we need to do is add some Tomcat instances:

sudo ./createInstance.sh 1 1
sudo ./createInstance.sh 1 2

These commands will create two instances (1 & 2) on server 1 - the first argument being the server number, the second argument the instance number. You can find the instances at $CR/instance_1 and $CR/instance_2. In this example, I'm only creating a single server cluster, but you can run the scripts on multiple servers to create a multi-server cluster - just make sure you use a different server number on each one. As a side note, the scripts only work if you use single digit numbers for the instance numbers, so don't try to create more than nine instances on a server with them!

Before we start the Tomcat instances, we must first deploy our application and configure it. For the physical deployment, we can use the deploy.sh script:

sudo ./deploy.sh /path/to/myapp-0.1.war

This will unpack the WAR into the $CR/shared/webapps/ROOT directory, so the application will be served from the root context. This makes it easier to load balance the application with something like Apache httpd. The next step is to configure the database and logging for our instances. If you look in the $CR/shared/lib directory, you will see a my-cluster-app-config.groovy. What's this for? If you cast your mind back to earlier in the article, you will remember that we added the line

grails.config.locations = [ "file:./${appName}-config.groovy", "classpath:${appName}-config.groovy" ]

to the application's Config.groovy file. Since the shared/lib directory is on the classpath of all the cluster instances, we can add extra configuration via this my-cluster-app-config.groovy file. This could include database connection settings, logging configuration, or anything else. In this case, we'll configure the database connection settings and make sure that log files are created per instance. Open up the file up and add the following:

dataSource {
    dbCreate = 'update'
    username = 'grails'
    password = 'grails'
    driverClassName = 'com.mysql.jdbc.Driver'
    dialect = org.hibernate.dialect.MySQL5InnoDBDialect
    url = 'jdbc:mysql://localhost/myapp'
}
 
log4j = {
    appenders {
        rollingFile name: 'main', file: "${System.getProperty('catalina.base')}/logs/main.log"
    }
 
    root {
        info 'main'
    }
}

On the database side, the above configures all instances to use the 'myapp' MySQL database with the 'grails' user. Of course, you need the MySQL JDBC driver on the classpath, but that's as easy as copying the JAR into the shared/lib directory. On the logging side, we want to make sure that the application log file is created in the logs directory of each instance. Since the location of the instance directory is available in the 'catalina.base' system property, we can easily include it in the log file's path as we have done above.

Assuming that the MySQL database already exists, we're ready to test the cluster. Start the Tomcat instances with these commands:

sudo ./run.sh start 1
sudo ./run.sh start 2

After a little while you should be able to point your browser at http://localhost:8091/home/arrive and see the message "You have arrived - counter initialised". Now replace the 'arrive' in the URL with 'counter' and reload the page a few times. The counter will increment on each refresh. So that's the first Tomcat instance working. What about session replication? To test that, go to this URL: http://localhost:8092/home/counter - you are now hitting the second server. If everything is working properly, you should see the counter incremented from the previous number you saw - the session replication is working!

The next step would be to put a load balancer in front of the Tomcat cluster, but that's beyond the scope of this article. Suffice it to say that you don't have to do anything special for Grails applications. Should you choose to use Apache httpd with mod_proxy_balancer, you may be interested in the modproxybalancer Grails plugin which allows you to redeploy your application via a single Grails command, although you need to have Tomcat's manager application installed in all your instances.

Wrap up

Since Grails applications are packaged as WAR files, you can deploy them to a servlet container just as you would any other Java web application. That's one of the key strengths of Grails. Tomcat clusters are no different, but they do force you to do a bit more custom configuration of a Grails application than would normally be the case. As you've seen, this is straightforward using externalised configurations. When you need more flexibility, you can always make use of the Grails build system with its events or use JNDI.

I hope this post proves useful to both Grails developers who have never set up a Tomcat cluster and those people who have to deploy Grails applications in production systems. Although their configuration is different to typical Java web applications, once you know the techniques, it's pretty straightforward.

As a final note, I have made the sample application available to download.

I'm the Grails Advocate at SpringSource (a division of VMware) and one of the core committers of the project. I have been using Grails since 2006 and have contributed several plugins on top of the work I have done on the core. These days I write about the framework and guide people through basic and advanced usage.

Comments

Complexity

Grails was created to ease the Java web development, but it looks like deployment to cluster was not simplified. Sounds quite complex to me...

Do you have the script on Windows?

It's very useful!
I need to create a tomcat cluster in Windows,do you have a Windows version of tomcat-cluster-scripts.zip?If have,would you like share it with us?

an error in Windows!

I change the script of the sample.And config two instance in Windows.Start up instance 1,all is OK.Then start up instance 2,there is an error:
2010-7-24 9:52:52 org.apache.catalina.ha.session.DeltaManager getAllClusterSessions
信息: Manager [localhost#]: skipping state transfer. No members active in cluster group.
So the session duplication is failed!
Why?

I have resolve the error in the last comment!

Thanks a lot!I have resolved the error:
Manager [localhost#]: skipping state transfer. No members active in cluster group
I will write a Chinese blog according your blog and publish it in www.groovyq.net.

@leshkanyc Without the

@leshkanyc Without the caching, you only have to create the template web.xml and add support for external configuration files. I wouldn't class that as particularly complex.

Creating the cluster is more involved and requires a fair bit of work unless you use something like the provided scripts. I'm not sure Grails could really do much here.

Could deploying to a Tomcat cluster be made easier still? Probably. I'm sure a plugin could do much of the work. There's already the plugin for automatic deployment to a mod_proxy_balancer cluster for example.

Nice!

Very nicely written article - thank you!

error at "sudo ./createCluster.sh my-cluster-app" command

Hj,when i run "sudo ./createCluster.sh my-cluster-app" command, i recieve message :
" Usage: createCluster.sh appName [cluster root] "
I don't know how solve it . Can u help me ? Thank !!

multiple ehcache.xml in war

Instead of replacing ehcache.xml during packing time, i keep the standalone and clustered ehcache configuration in the classpath and let a configuration property decide which to use:

in Config.groovy:

if (System.getProperty("clustered", "false") == "true") {
hibernate.config.location = ["classpath:hibernate.cluster.cfg.xml"]
}

This adds an additional hibernate config file in case of a clustered setting.

The file hibernate.cluster.cfg.xml in grails-app/conf/hibernate looks like this:
<!DOCTYPE hibernate-configuration SYSTEM
"http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
<session-factory>
<property name="net.sf.ehcache.configurationResourceName">/ehcache_distributed.xml</property>
</session-factory>
</hibernate-configuration>

This works, because EhCache is using the configuration property "net.sf.ehcache.configurationResourceName" if set to read its configuration from this URL instead of using classpath:ehcache.xml

This works with Grails 1.3.x (probably with newer Grails versions as well, but not yet tested it).

Post new comment

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.