There are three schools of thought on how to achieve the same task of managing a large production Tomcat infrastructure: distributed command-line solutions, custom tools using Java Management Extensions (JMX), and full-blown commercial software suites designed for managing Tomcat specifically. All of the methods and utilities for managing Tomcat have their own respective tradeoffs of elegance, simplicity, manageability, and cost.
The Tomcat Manager application is an excellent utility for managing Tomcat server. It has built-in support for deploying and un-deploying applications, reloading applications, gathering statistics about the JVM and Web application, as well as starting and stopping applications. IT even includes a Web-based JMX proxy where a user can modify and query MBeans through the manager application.
Where it falls short is that it primarily only supports functions for performing management operations at the servlet container level and above. It doesn’t allow a user to restart, start or stop the JVM that the Tomcat instance runs on from this Web interface and doesn’t have a built-in method for managing groups of Tomcat servers.
The simplest method is using ssh-keys and shell/Python/Ruby/Groovy scripts, which requires little effort to implement, but lacks in elegance. There are a number of service wrappers that support automatic restarts on failure, schedule restarts and taking thread dumps on failure, but they don’t come with a good interface through being managed remotely by a Java standard such as JMX.
Many institutions implement homegrown or scripted solutions for managing large Tomcat infrastructures. When dealing with anything beyond 10 Tomcat servers, it becomes tedious to log into each host and start and stop each Tomcat instance.
Many solutions exist for running a distributed command against a large group of Tomcat clusters. Some organizations set up SSH keys on all hosts and use a scripted or CGI Web interface to send a single command to a large group of servers; however, when dealing with thousands of hosts, installing the keys can become a tedious task, and without a configuration management utility, such as Puppet or Cfengine this can consume a large amount of time to perform.
Tentakel is a distributed command execution utility, and is primarily used for managing supercomputing clusters. Tentakel is written in Python, and is found in UCSD’s Rocks Clusters Supercomputing Platform. Tentakel also happens to come in handy for managing large amounts of Tomcat servers.
Tentakel is handy for managing large Tomcat infrastructures because of its ability to group servers together in a single configuration file, and run commands in parallel in groups. Tentakel uses ssh and rsh for it’s execution mechanism, and also requires that to be configured, so again a good configuration management utility or provisioning utility must be utilized in order to install the key on all hosts to be managed, but regardless of implementation some transport mechanism has to be configured on each destination Tomcat server. What is important is how effective and simple that is to perform. Once the transport mechanism is installed, be it ssh, rsh, or a custom agent, there still needs to be a central point of management that is easy to maintain.
I find that Tentakel’s use of a single configuration file for managing groups of Tomcat servers makes this easy to do. It also has a plug-in interface, which can be extended to implement custom commands for making use of the Tomcat management application’s facilities for managing Web applications in a Tomcat container.
For example, if I have two Tomcat clusters, each with 500 servers in each cluster, lets say east and west coast. I can restart all of my Tomcat servers on the West Coast in parallel, by executing a single command, and provided such an action wouldn’t take out my database or data store, then I could manage 500 Tomcat instances with a single command, provided they are all grouped in a single configuration file. Having the ability to manage large amounts of Tomcat servers from a single interface saves time through automation, and saving time in the enterprise is directly related to saving money and efficiency.
The Catalina MBean comprises of the majority of operations that can be invoked on a Web application running in a Tomcat container, except those for controlling the JVM itself. Tying all of the application management and monitoring pieces together along with controlling large amounts of Tomcat instances is ideal. However it requires using security on JMX, which in my experience many companies do not.
The option to accomplish managing the JVM through JMX is to write a custom MBean for stopping and restarting Tomcat. The only problem is once the JVM goes down, so does the JMX listener, so there isn’t a clear way to start up a Tomcat server via JMX without an agent listener process.
There are a few agent based models for administering Tomcat application servers. Agent based models place a daemon process directly on the Tomcat server that is managed remotely and always listening for control commands such as starting, stopping, and deploying applications. This is an efficient method for managing Tomcat; only a few solutions exist.
Java Service Wrapper, and Yet Another Java Service Wrapper are utilities that allow for daemon-izing your Java processes on a Unix based system, or running it as a service on a Windows system, which gives it additional functionality such as automatic and scheduled restarts and JVM start up option configuration, but they do not have a good facility for securely and remotely restarting the JVM or remotely pushing configurations.
Hyperic HQ and MuleSoft Tcat Server are agent-based solutions with out of the box for managing large Tomcat infrastructures. As long as the agent is installed it can automatically detect running Tomcat instances and manage them from a single Web dashboard or command-line interface out-of-the-box.
The other option is to create an agent that always listens on the machine where the Tomcat server is hosted, and listen on a given port and send commands for invoking and shutting down processes. This still requires deploying a custom agent and creating a management application to keep an inventory of all managed servers.
There are many solutions available for Managing large Tomcat infrastructures available, although only a few that are well designed for both managing Tomcat servers at a process level, as well as managing application deployments. Command line utilities are the simple, cost-efficient and require the least amount of tooling time to start managing large Tomcat infrastructures. They are not elegant, but with tools like Tentakel, there is a central configuration point that keeps a level of sanity to effectively manage large Tomcat infrastructures.
There are some service wrappers for Java processes that allow for daemon-izing or turning a Java process into a configurable service. These service wrappers add a nice additional layer of control to the process of the Tomcat server, but don’t have well-developed facilities for managing servers remotely. This is something that must be built on top of these services or purchased.
Depending on the needs for managing a large Tomcat implementation, I have presented a solution from the very basic and effective, to the more elegant and thorough methods. Most users will find that some combination of all methods suits their needs. Some readers will want to go with a commercial solution if time and support is more of an issue than cost.