TomcatExpert

How to Scale Apache Tomcat in the Cloud with RabbitMQ and JMX

posted by jbrisbin on June 3, 2010 10:20 PM

I sometimes pine for the days when I just had one server to worry about. I wax nostalgic, remembering how easy my life was when I didn't have servers and virtual machines growing out of my ears. It's almost the same feeling I get thinking about the days when I only had one child and you could just pop them in a carseat and take off. Now I've got kids driving themselves and their siblings to several activities a day and I can hardly keep track of whether I'm coming or going. I feel the same way about my data center. It's all grown up now and, while it may not have its own driver's license, I can still lose sleep over it getting a little more out of my control every day.

Such are the demands of the uber-modern data center. Deploying your applications into the cloud and keeping them in synch can cause one to either swear profusely or put in for a month of vacation. Whether you're in a virtual or hybrid private cloud, or working on one of the big-name cloud providers, keeping track of what services are available and managing them once they're up is a common requirement. I approached this problem in our own private cloud by writing a Tomcat LifecycleListener that hooks into our RabbitMQ servers to keep interested queue subscribers updated with the internal state of our SpringSource tcServer instances, as well as providing the ability to invoke JMX MBeans via asynchronous messaging. Since this system uses AMQP, any language that has an AMQP client that can talk to RabbitMQ can invoke JMX-managed MBeans. RabbitMQ is the backbone of our private cloud. Our tcServer instances use it to communicate with one another for session clustering and custom management tools use it to facilitate deployment artifact rollouts. We use messaging as a one-size-fits-all communication bus. Java can talk to Ruby can talk to Python can talk to C. Setting up a RabbitMQ server is beyond the scope of this discussion. If you're running Ubuntu 10.04, simply issue a "sudo apt-get install rabbitmq-server". If you just want to test this scenario, you can run a local RabbitMQ server very easily. I'll leave that as an exercise to the reader.

The Tomcat Listener

To relay internal Lifecycle events to the cloud, I wrote a Listener that I install in the tcServer instances I want to manage. It's part of the virtual cloud utilities I've opensourced and released.Configuration is fairly straightforward, though it might get a little confusing with all the exchange and queue names. First, the relevant snippage from server.xml:

<a a="" classname="com.jbrisbin.vcloud.mbean.CloudInvokerListener" eventsexchange="vcloud.events" github.com="" href="http://www.springsource.com/products/tcserver?p=tcserver&lp=1&cid=70180000000wVJj>SpringSource tcServer</a> instances, as well as providing the ability to invoke JMX MBeans via asynchronous messaging. Since this system uses AMQP, any language that has an AMQP client that can talk to RabbitMQ can invoke JMX-managed MBeans.</p> <!--{cke_protected}{C}%3C!%2D%2Dbreak%2D%2D%3E--> <h2> The RabbitMQ Server</h2> <p> RabbitMQ is the backbone of our private cloud. Our tcServer instances use it to communicate with one another for session clustering and custom management tools use it to facilitate deployment artifact rollouts. We use messaging as a one-size-fits-all communication bus. Java can talk to Ruby can talk to Python can talk to C. Setting up a RabbitMQ server is beyond the scope of this discussion. If you're running Ubuntu 10.04, simply issue a "sudo apt-get install rabbitmq-server". If you just want to test this scenario, you can run a local RabbitMQ server very easily. I'll leave that as an exercise to the reader.</p> <h2> The Tomcat Listener</h2> <p> To relay internal Lifecycle events to the cloud, I wrote a Listener that I install in the tcServer instances I want to manage. It's part of the virtual cloud utilities I've opensourced and released on GitHub: <a data-cke-saved-href=" http:="" instancename="tcserver.${hostname}.${instance.id}" jbrisbin="" listener="" master="" mbean-invoker="" mbeaneventsexchange="vcloud.mbean" mbeaneventsqueue="mbean.${hostname}.${instance.id}" mbeaneventsroutingkey="tcserver.${hostname}.${instance.id}" mqhost="localhost" mqpassword="guest" mqport="5672" mquser="guest" mqvirtualhost="/" tree="" vcloud="">To differentiate the tcServer instances from one another in the cloud, I use convention over configuration. On each Ubuntu virtual machine that runs tcServer, we've configured the instances with names that are unique to the VM. Each VM has a unique hostname which is derived from the IP address, so a cloud-unique name for a given tcServer instance is a combination of hostname and instance name.</a>

SpringSource tcServer instances, as well as providing the ability to invoke JMX MBeans via asynchronous messaging. Since this system uses AMQP, any language that has an AMQP client that can talk to RabbitMQ can invoke JMX-managed MBeans.

The RabbitMQ Server

RabbitMQ is the backbone of our private cloud. Our tcServer instances use it to communicate with one another for session clustering and custom management tools use it to facilitate deployment artifact rollouts. We use messaging as a one-size-fits-all communication bus. Java can talk to Ruby can talk to Python can talk to C. Setting up a RabbitMQ server is beyond the scope of this discussion. If you're running Ubuntu 10.04, simply issue a "sudo apt-get install rabbitmq-server". If you just want to test this scenario, you can run a local RabbitMQ server very easily. I'll leave that as an exercise to the reader.

The Tomcat Listener

To relay internal Lifecycle events to the cloud, I wrote a Listener that I install in the tcServer instances I want to manage. It's part of the virtual cloud utilities I've opensourced and released on GitHub: <a data-cke-saved-href=" http:="" jbrisbin="" master="" mbean-invoker="" the="" tree="" vcloud="">Lifecycle events are published to a topic exchange, the name of which is configured with the "eventsExchange" setting. There's no magic here at all. Clients that wish to be notified of Tomcat Lifecycle events simply need to bind their queue to this exchange with a routing key apropos to the events they're interested in. The names for these events are in the org.apache.catalina.Lifecycle class (check the Tomcat source distribution). The event names for Tomcat 6.0 are:

Jon Brisbin is an Architect/Analyst/Java Guru at NPC International, the world's largest Pizza Hut franchisee. He's been deploying web applications on Tomcat for over 10 years and currently focuses on cloud computing (virtual, hybrid, and private). He built a private cloud from scratch using VMware ESX, Ubuntu Linux, packing tape, and rusty baling wire. He's done consulting work with industry leaders and Mom-and-Pops alike. Prior to NPC, Jon developed new application frameworks, integrated the AS/400 with UNIX and Windows systems, developed Lotus Domino applications, hacked websites together with Perl CGI and a text editor, and served with US Air Force Intelligence in a very hot and sandy, but undisclosed, location. He lives in the rural Midwest.

He blogs on Web 2.0 (and sundry topics) on his website: http://jbrisbin.com/web2

Comments

Interesting..

Sorry, I misposted - comment moved to http://www.tomcatexpert.com/blog/2010/06/21/clustering-cloud-friendly-tomcat-sessions-rabbitmq-part-i

Post new comment

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.