How to Scale Apache Tomcat in the Cloud with RabbitMQ and JMX

posted by jbrisbin on June 3, 2010 10:20 PM

I sometimes pine for the days when I just had one server to worry about. I wax nostalgic, remembering how easy my life was when I didn't have servers and virtual machines growing out of my ears. It's almost the same feeling I get thinking about the days when I only had one child and you could just pop them in a carseat and take off. Now I've got kids driving themselves and their siblings to several activities a day and I can hardly keep track of whether I'm coming or going. I feel the same way about my data center. It's all grown up now and, while it may not have its own driver's license, I can still lose sleep over it getting a little more out of my control every day.

Such are the demands of the uber-modern data center. Deploying your applications into the cloud and keeping them in synch can cause one to either swear profusely or put in for a month of vacation. Whether you're in a virtual or hybrid private cloud, or working on one of the big-name cloud providers, keeping track of what services are available and managing them once they're up is a common requirement. I approached this problem in our own private cloud by writing a Tomcat LifecycleListener that hooks into our RabbitMQ servers to keep interested queue subscribers updated with the internal state of our SpringSource tcServer instances, as well as providing the ability to invoke JMX MBeans via asynchronous messaging. Since this system uses AMQP, any language that has an AMQP client that can talk to RabbitMQ can invoke JMX-managed MBeans. RabbitMQ is the backbone of our private cloud. Our tcServer instances use it to communicate with one another for session clustering and custom management tools use it to facilitate deployment artifact rollouts. We use messaging as a one-size-fits-all communication bus. Java can talk to Ruby can talk to Python can talk to C. Setting up a RabbitMQ server is beyond the scope of this discussion. If you're running Ubuntu 10.04, simply issue a "sudo apt-get install rabbitmq-server". If you just want to test this scenario, you can run a local RabbitMQ server very easily. I'll leave that as an exercise to the reader.

The Tomcat Listener

To relay internal Lifecycle events to the cloud, I wrote a Listener that I install in the tcServer instances I want to manage. It's part of the virtual cloud utilities I've opensourced and released.Configuration is fairly straightforward, though it might get a little confusing with all the exchange and queue names. First, the relevant snippage from server.xml:

<a a="" classname="com.jbrisbin.vcloud.mbean.CloudInvokerListener" eventsexchange="vcloud.events" github.com="" href="http://www.springsource.com/products/tcserver?p=tcserver&lp=1&cid=70180000000wVJj>SpringSource tcServer</a> instances, as well as providing the ability to invoke JMX MBeans via asynchronous messaging. Since this system uses AMQP, any language that has an AMQP client that can talk to RabbitMQ can invoke JMX-managed MBeans.</p> <!--{cke_protected}{C}%3C!%2D%2Dbreak%2D%2D%3E--> <h2> The RabbitMQ Server</h2> <p> RabbitMQ is the backbone of our private cloud. Our tcServer instances use it to communicate with one another for session clustering and custom management tools use it to facilitate deployment artifact rollouts. We use messaging as a one-size-fits-all communication bus. Java can talk to Ruby can talk to Python can talk to C. Setting up a RabbitMQ server is beyond the scope of this discussion. If you're running Ubuntu 10.04, simply issue a "sudo apt-get install rabbitmq-server". If you just want to test this scenario, you can run a local RabbitMQ server very easily. I'll leave that as an exercise to the reader.</p> <h2> The Tomcat Listener</h2> <p> To relay internal Lifecycle events to the cloud, I wrote a Listener that I install in the tcServer instances I want to manage. It's part of the virtual cloud utilities I've opensourced and released on GitHub: <a data-cke-saved-href=" http:="" instancename="tcserver.${hostname}.${instance.id}" jbrisbin="" listener="" master="" mbean-invoker="" mbeaneventsexchange="vcloud.mbean" mbeaneventsqueue="mbean.${hostname}.${instance.id}" mbeaneventsroutingkey="tcserver.${hostname}.${instance.id}" mqhost="localhost" mqpassword="guest" mqport="5672" mquser="guest" mqvirtualhost="/" tree="" vcloud="">To differentiate the tcServer instances from one another in the cloud, I use convention over configuration. On each Ubuntu virtual machine that runs tcServer, we've configured the instances with names that are unique to the VM. Each VM has a unique hostname which is derived from the IP address, so a cloud-unique name for a given tcServer instance is a combination of hostname and instance name.</a>

SpringSource tcServer instances, as well as providing the ability to invoke JMX MBeans via asynchronous messaging. Since this system uses AMQP, any language that has an AMQP client that can talk to RabbitMQ can invoke JMX-managed MBeans.

The RabbitMQ Server

RabbitMQ is the backbone of our private cloud. Our tcServer instances use it to communicate with one another for session clustering and custom management tools use it to facilitate deployment artifact rollouts. We use messaging as a one-size-fits-all communication bus. Java can talk to Ruby can talk to Python can talk to C. Setting up a RabbitMQ server is beyond the scope of this discussion. If you're running Ubuntu 10.04, simply issue a "sudo apt-get install rabbitmq-server". If you just want to test this scenario, you can run a local RabbitMQ server very easily. I'll leave that as an exercise to the reader.

The Tomcat Listener

• before_start
• after_start
• SpringSource tcServer instances, as well as providing the ability to invoke JMX MBeans via asynchronous messaging. Since this system uses AMQP, any language that has an AMQP client that can talk to RabbitMQ can invoke JMX-managed MBeans.</p> <!--{cke_protected}{C}%3C!%2D%2Dbreak%2D%2D%3E--> <h2> The RabbitMQ Server</h2> <p> RabbitMQ is the backbone of our private cloud. Our tcServer instances use it to communicate with one another for session clustering and custom management tools use it to facilitate deployment artifact rollouts. We use messaging as a one-size-fits-all communication bus. Java can talk to Ruby can talk to Python can talk to C. Setting up a RabbitMQ server is beyond the scope of this discussion. If you're running Ubuntu 10.04, simply issue a "sudo apt-get install rabbitmq-server". If you just want to test this scenario, you can run a local RabbitMQ server very easily. I'll leave that as an exercise to the reader.</p> <h2> The Tomcat Listener</h2> <p> To relay internal Lifecycle events to the cloud, I wrote a Listener that I install in the tcServer instances I want to manage. It's part of the virtual cloud utilities I've opensourced and released on GitHub: <a data-cke-saved-href=" http:="" jbrisbin="" master="" mbean-invoker="" tree="" vcloud="">stop
• destroy
• For the purposes of this article, we'll look at some Ruby code that listens for "after_start" events because it deals with application server cluster membership.

  The Ruby Code

  The Subscribers

  # Create, bind, and subscribe to a test queue for the results of JMX operation. sub.queue('test.ruby.results', :auto_delete => true).bind('').subscribe do |headers, body| data = JSON.load(body) data['listApplications'].each do |v| if v =~ /^\// puts "app: #{v}" end end end

  The events subscriber is pretty straightforward as well:

  <a .subscribe="" :auto_delete=">" :routing_key=">" a="" and="" do="" end="" event="" for="" getting="" github.com="" href="http://www.springsource.com/products/tcserver?p=tcserver&lp=1&cid=70180000000wVJj>SpringSource tcServer</a> instances, as well as providing the ability to invoke JMX MBeans via asynchronous messaging. Since this system uses AMQP, any language that has an AMQP client that can talk to RabbitMQ can invoke JMX-managed MBeans.</p> <!--{cke_protected}{C}%3C!%2D%2Dbreak%2D%2D%3E--> <h2> The RabbitMQ Server</h2> <p> RabbitMQ is the backbone of our private cloud. Our tcServer instances use it to communicate with one another for session clustering and custom management tools use it to facilitate deployment artifact rollouts. We use messaging as a one-size-fits-all communication bus. Java can talk to Ruby can talk to Python can talk to C. Setting up a RabbitMQ server is beyond the scope of this discussion. If you're running Ubuntu 10.04, simply issue a "sudo apt-get install rabbitmq-server". If you just want to test this scenario, you can run a local RabbitMQ server very easily. I'll leave that as an exercise to the reader.</p> <h2> The Tomcat Listener</h2> <p> To relay internal Lifecycle events to the cloud, I wrote a Listener that I install in the tcServer instances I want to manage. It's part of the virtual cloud utilities I've opensourced and released on GitHub: <a data-cke-saved-href=" http:="" instance="headers.properties[:routing_key].split('.')" jbrisbin="" lifecycle="" master="" mbean-invoker="" notifications.="" qname="mbean.#{hostname}.#{instance}" queue="" subscribe="" to="" tree="" vcloud="">The routing key is a three-part key. It contains the Lifecycle event name, the hostname of the server, and the tcServer instance name that emitted the event, all separated by periods. We simply split this key and construct a queue name back to that server specifically. If we were wanting to send a message out to all tcServers bound to this exchange, we'd publish to the exchange and use a routing key instead.</a>

  # Call the tcServer JMX MBean to list installed applications. def list_applications(qname) data = { "mbean" => "tcServer:type=Serviceability,name=Deployer", "operation" => "listApplications", "parameters" => [ [ "java.lang.String", "Catalina" ], [ "java.lang.String", "localhost" ] ] } json = JSON.dump(data) puts "JSON: #{json}" MQ.new.queue(qname).publish(json, :reply_to => 'test.ruby.results') end

  It's All Downhill From Here

  The biggest issue I see with adopting anything like a cloud architecture is that task-oriented virtual machines can breed like rabbits. You quickly end up with virtual machines named "tomcat1", "tomcat2", ... "tomcat12" ad nauseum. Managing these servers using asynchronous messaging makes a lot of sense. Thanks to the way queue subscriptions work, you don't have to maintain lists of active connections to which you need to replicate data. Using routing keys that incorporate event names and hostnames means your AMQP clients can listen for only those events in which they're interested. One could partition hundreds of servers using a sane combination of RabbitMQ virtual hosts, exchanges, and routing keys.

  <a href="http://www.springsource.com/products/tcserver?p=tcserver&lp=1&cid=70180000000wVJj>SpringSource tcServer</a> instances, as well as providing the ability to invoke JMX MBeans via asynchronous messaging. Since this system uses AMQP, any language that has an AMQP client that can talk to RabbitMQ can invoke JMX-managed MBeans.</p> <!--{cke_protected}{C}%3C!%2D%2Dbreak%2D%2D%3E--> <h2> The RabbitMQ Server</h2> <p> RabbitMQ is the backbone of our private cloud. Our tcServer instances use it to communicate with one another for session clustering and custom management tools use it to facilitate deployment artifact rollouts. We use messaging as a one-size-fits-all communication bus. Java can talk to Ruby can talk to Python can talk to C. Setting up a RabbitMQ server is beyond the scope of this discussion. If you're running Ubuntu 10.04, simply issue a &quot;sudo apt-get install rabbitmq-server&quot;. If you just want to test this scenario, you can run a local RabbitMQ server very easily. I'll leave that as an exercise to the reader.</p> <h2> The Tomcat Listener</h2> <p> To relay internal Lifecycle events to the cloud, I wrote a Listener that I install in the tcServer instances I want to manage. It's part of the virtual cloud utilities I've opensourced and released on GitHub: <a data-cke-saved-href=" http:="" github.com="" jbrisbin="" vcloud="" tree="" master="" mbean-invoker="" "="">I'm excited by the opportunities Tomcat users have to deploy their applications into private, hybrid cloud architectures. Lightweight is the new black and Tomcat, RabbitMQ, and asynchronous messaging play right into that hand like the Jack of Spades.

Jon Brisbin is an Architect/Analyst/Java Guru at NPC International, the world's largest Pizza Hut franchisee. He's been deploying web applications on Tomcat for over 10 years and currently focuses on cloud computing (virtual, hybrid, and private). He built a private cloud from scratch using VMware ESX, Ubuntu Linux, packing tape, and rusty baling wire. He's done consulting work with industry leaders and Mom-and-Pops alike. Prior to NPC, Jon developed new application frameworks, integrated the AS/400 with UNIX and Windows systems, developed Lotus Domino applications, hacked websites together with Perl CGI and a text editor, and served with US Air Force Intelligence in a very hot and sandy, but undisclosed, location. He lives in the rural Midwest.

He blogs on Web 2.0 (and sundry topics) on his website: http://jbrisbin.com/web2