TomcatExpert

A Finer Point of Apache Tomcat Valves

posted by chanthing on November 10, 2011 10:09 AM

1. Introduction

Valves have been an integral feature of Apache Tomcat since version 4 was introduced over seven years ago. As their name suggests, valves provide a way of inserting functionality within a pipeline, in this case, the Tomcat request / response stream. One simply writes a subclass of org.apache.catalina.valves.ValveBase or a class that implements the org.apache.catalina.valves.Valve interface and then adds an XML element with the valve’s classname to the appropriate configuration file (in most classes as a sub element of Engine in server.xml). Then, at some point (we’ll come back to that) in the processing of a request, your valve’s invoke method will be called. The invoke method gets passed both the Request and the Response objects, and is free to do whatever it likes with them (though having the power doesn’t mean it’s a good idea to use it).

You may be familiar with this paradigm through servlet filters used by web applications to do application-specific processing of the request / response pipeline. The key distinction between servlet filters and Tomcat Valves is that Valves are applied and controlled through the configuration of the application server. Depending on the container definition where the Valve element appears in the Tomcat configuration, the valve could be configured for all applications on the application server, a subset of applications, or even a single application (by locating the Valve element within a given Context).

This is a simple powerful model that has been written about extensively. A Google search on “tomcat valve” turns up a multitude of descriptions, examples, and “how tos”. The Reference Page on “The Valve Component” that ships with Apache Tomcat 7 documents the mechanism thoroughly along with descriptions of the valve implementations that ship by default. So why yet another article on the subject? What do I hope to add to the canon?

The effort started simply enough: The plan was to demonstrate the configuration and use of the ThreadDiagnosticsValve that ships with VMware’s vFabric tc Server, a commercial application server based on Apache Tomcat. Additionally I would write and configure a custom Valve, in this case, a valve to exercise tc Server’s ThreadDiagnosticsValve so that I could demonstrate its use and effects without actually having to have a misbehaving application to trigger it. Not exactly Nobel material, but I thought it would be a useful adjunct to the existing documentation and an interesting exercise.

However, it didn’t go exactly as planned and looking at the reason it didn’t is probably as helpful in understanding Tomcat Valves as the original exercise.

2. Valve Crash Course

As mentioned above, there are many sources for basic information on Apache Tomcat Valves. The reference page (http://tomcat.apache.org/tomcat-7.0-doc/config/valve.html) is one of the best, but I’ll still do a quick review here so you have the basics. A valve is a component that gets inserted in the request processing pipeline for a given Catalina container (Engine, Host, or Context). This is done by adding an equivalent XML Valve element to a structure in the appropriate configuration file. The class attribute of the element identifies the type of valve to be inserted. Other attributes of the Valve element provide values for the valve’s parameters. Apache Tomcat comes with a number of valves, including AccessLogValve, RemoteAddressFilter, SingleSignOnValve, and others.

2.1 Example Valve Configuration

RemoteAddressFilter allows you to specify a regular expression that must match the IP address of the requesting client in order for the request to be processed. To insert this valve in your default engine’s pipeline, you would place the following XML as the first element nested inside your configuration’s Engine element like so:

	<engine name="Catalina">
 
<valve classname="org.apache.catalina.valves.RemoteAddrValve">allow="127\.0\.0\.1"/></valve></engine>

The allow attribute is a regular expression, not a string or wildcard string. In Tomcat 6, it’s a comma-separated list of regexp.

This example uses ipv4 format, though ipv6 is supported as well. (And don’t be fooled by the fact that the class name is RemoteAddrValve and not RemoteAddressFilter. Most valves have the same name as their implementing class.) With this valve in place, any request originating from a system other than localhost (the host running Apache Tomcat) will get denied. This idiom is frequently used on application servers that are front- ended by Apache Web Servers running on the same host, or on a different, well-known host, or even (using wild- cards) a subnet where the proxy servers are known to reside. Obviously this is more secure than being open to the world. This is exactly the type of pipeline transformation that should be global to the application server, and hence implemented as a valve, rather than be restricted to a particular web application like a filter would be.

As stated before, Valve elements may also be nested inside Host and Context elements in order to more specifically define the pipeline in which the valve will be inserted.

3. ThreadDiagnosticsValve

vFabric tc Server, VMware’s Apache Tomcat-based application server provides a valve to monitor application requests / responses and periodically report information on these requests, including the URL of the problematic request, in order to allow investigation and troubleshooting of “slow” applications. If you like you can even configure it to report information concerning the effect of database access on the performance.

See http://pubs.vmware.com/vfabric5/index.jsp?topic=/com.vmware.vfabric.tc-server.2.6/admin/manual-thread-diagnostics.html for documentation on enabling and configuring this valve. The valve is designed to work in cooperation with vFabric Hyperic (the management and monitoring tool for tc Server), so that you will receive alerts in Hyperic when an application is running slowly, but to keep our example simple we will skip that part of the configuration; as we set it up, the valve definition we add to the server configuration in server.xml will cause the value to print periodic slow request reports to the tc Server log file (catalina.date.log). In a production environment, these log messages would be detected by vFabric Hyperic and turned into alerts on the Hyperic console.

VMware contributed back this idea of a thread diagnostics valve into Apache Tomcat through the StuckThreadDetectionValve.

4. RandomDelayValve

I wanted a simple way to demonstrate the capabilities of the ThreadDiagnosticsValve. Ironically, it’s not always easy to make a real application deliberately go slow, so I decided to write a valve of my own, one that would make requests take longer by sleeping in the valve’s invoke method. I wrote RandomDelayValve, a simple valve that adds sleep calls of varying lengths to a specified percentage of requests.

The invoke method for the RandomDelayValve

/**
     *   If selected as a "slow" request (via random number and percentage parameter),
     *   add a certain amount of delay to simulate a long-running request.
     */
    public void invoke(Request request, Response response) throws IOException,
                    ServletException {
        /* Use percentage to find out if this request is delayed. */
        boolean willBeDelayed = (random.nextInt(100) > percentDelayed);
        if (willBeDelayed) {
            int delay;
            delay = random.nextInt(maxDelay - minDelay) + minDelay;
            /* Report delay */
            if (containerLog.isInfoEnabled()) {
                containerLog.info("Delaying request " + request.getDecodedRequestURI()
                                        + " for " + delay + " ms.");
            }
            /* Delay request by sleeping */
            try {
                Thread.sleep(delay);
            } catch (InterruptedException ie) {
                containerLog.error("Got an exception while sleeping.");
            }
        }
        /* Invoke next valve */
        getNext().invoke(request, response);
    }

 

One key thing to note is the last line of our invoke method, which continues the processing of the request-response pipeline by calling the invoke method of the next element in the pipeline. Omitting that line would cause processing of the pipeline to stop and no result would be returned. (Take note, this will shortly be relevant.)

 

4.1 Getting, Building, and Installing the Valve

The complete source code for RandomDelayValve, along with an ant build file is available on github at (https://github.com/chanthing/RandomDelayValve/zipball/master).

To build and install (assumes you have ant and javac correctly configured and in your path):

  1. Download the tar file (chanthing-RandomDelayValve-3815c7a.zip)
  2. unzip chanthing-RandomDelayValve-3815c7a.zip
  3. cd chanthing-RandomDelayValve-3815c7a.zip
  4. ant
  5. cp RandomDelayValve.jar tcServerRoot/instance/lib where tcServerRoot is the directory in which you've installed tc Server and instance is the CATALINA_BASE directory of the instance you are configuring.

4.2 Configuring tc Server to use the two valves

Now that you have the code implementing the two jars available in your tc Server instance (ThreadDiagnosticsValve is in tcServerRoot/tomcat-7.0.12.A.RELEASE/lib/tcServer.jar, depending on what version you have installed), you can add the two valves to your tc Server configuration file (instancedir/conf/server.xml).

Fragment of server.xml that configures the two valves

<Service name="Catalina">
        <Executor maxThreads="300"
                  minSpareThreads="50"
                  name="tomcatThreadPool"
                  namePrefix="tomcat-http--"/>
        <Engine defaultHost="localhost"
                name="Catalina">
            <Valve className="vmware.stg.demo.tcvalve.RandomDelayValve"
                   percentDelayed="75"
                   maxDelay="10000"
                   minDelay="5000"/>
     <Valve           className="com.springsource.tcserver.serviceability.request.ThreadDiagnosticsValve"
                        threshold="5000"
                        history="1500"
                        notificationInterval="60000"
                        loggingInterval="100000"
                        logExtendedData="true"/>

As noted earlier, the values of an xml attributes of the Valve element sets the value of the corresponding parameter. For instance, for the ThreadDiagnosticsValve, threshold indicates the number of milliseconds the processing of a request must exceed in order to be reported, in this case, 1500. The other attributes control the recording and reporting of the violating requests. Note that the valve has two reporting mechanisms: one where it simply writes the information to the tc Server log, the other where it notifies via JMX. So for instance, notificationInterval controls the frequency of notifications via JMX, whereas loggingInterval does the same for the reports that get written to the logfile. For RandomDelayValve, percentDelayed specifies the percentage of requests that will have a sleep inserted in order to delay them; minDelay and maxDelay indicate the range (in milliseconds of the delay that will get applied.

So this all looks copacetic, right? We’ve got our RandomDelayValve delaying requests and then the ThreadDiagnosticsValve measuring response time and reporting on requests that exceed the response threshold.

Incidentally, for those of you who don’t have tc Server, and hence don’t have ThreadDiagnosticsValve, you can grab the source for a valve that reports request performance from a set of examples written by Nicolas Frankel. These accompany his article Tomcat’s Valve, an alternative to Filter.

The valve in question is PerformanceValve, an extremely simplified version, but one that will report the response time for particular requests.

So, we have our instance configured so that our valve will cause some percentage of application requests to be delayed by an amount that should trigger the reporting of the ThreadDiagnosticsValve. We start are instance and click away on any application deployed on that instance, and we can see in the catalina log file messages like:

 06-Nov-2011 22:56:19.731 INFO vmware.stg.demo.tcvalve.RandomDelayValve.invoke Delaying request / for 6223 ms. t

This shows us that our filter is indeed working and causing requests to be delayed. We expect that once the amount of time specified in the loggingInterval attribute of ThreadDiagnosticValve elapses, we will get a report on slow running requests from that valve. However, we patiently click and make requests, verifying that they are delayed by an amount greater than the diagnostic valve’s threshold and yet never see the report we are expecting to be written to the log file. What is wrong?

Remember that we are dealing with a pipeline; our valves are added to the pipeline in an order and then executed in the same order. In fact, one major responsibility of a valve’s is that after it has done its work, it must call the next valve in the pipeline via the line: 

getNext().invoke(request, response);

Because of the order the valves are specified in server.xml, in our example, that line in RandomDelayValve calls the invoke method of ThreadDiagnosticsValve,

So by the time the ThreadDiagnosticsValve has a chance to start measuring performance, we have already done our sleep and the request proceeds normally.

An important lesson has been learned: the order in which our valves are added to the pipeline is significant. This seems trivially obvious after a little thought, but when you are in a rush to start exercising an exciting new feature, it is easy to lose sight of the details, no matter how obvious. The solution in our case is to change the order of the valves in server.xml, so that the pertinent fragment now looks like

Correct fragment of server.xml that configures the two valves

<Service name="Catalina">
        <Executor maxThreads="300"
                  minSpareThreads="50"
                  name="tomcatThreadPool"
                  namePrefix="tomcat-http--"/>
        <Engine defaultHost="localhost"
                name="Catalina">
 
            <Valve className="com.springsource.tcserver.serviceability.request.ThreadDiagnosticsValve"
                        threshold="5000"
                        history="1500"
                        notificationInterval="60000"
                        loggingInterval="100000"
                        logExtendedData="true"/>
<Valve className="vmware.stg.demo.tcvalve.RandomDelayValve"
                   percentDelayed="75"
                   maxDelay="10000"
                   minDelay="5000"/>

5. Conclusion

The key lesson here is not to take details for granted. One must always consider what operations are actually taking place behind the scenes. In the case of Tomcat request processing chains and Valves, we saw that at at least one point of significance is the order in which the valves are configured. Some further investigation in how Tomcat uses request pipelines (remember that the pipeline is responsible for the eventual invocation of the actual servlet code) may reveal some other subtleties that are worth further discussion.

Channing Benson has over 20 years of experience in the computer industry. Throughout his tenure he has helped customers adopt and master developing technologies, starting with  X11/Motif on Unix-based workstations, cross-Unix porting and C language optimization, and Linux development and performance tuning. While working at HP, he contributed to the FOSSology Open Source project. Channing has been a Java programmer/architect since its genesis in 1995, working primarily to help customers and independent software vendors optimize their programs for the latest available platforms, from EJBs and J2EE to Spring on lightweight containers like Tomcat and tc Server, and now cloud hosts like Cloud Foundry and high performance data grids like GemFire.

 

Comments

Why?

I thought valves were being phased out, and that their use should not be encouraged... use javax.servlet.Filter instead!!

Valves vs filters

If you're an application developers, then for sure use filters where possible. When doing internal things inside Tomcat, like integration at lower levels, Valves is the way to go.
The Apache Tomcat dev group looked to convert all valves to filters, but it turned out it would require a lot of work making the filters get access to internal components of Tomcat. This would also possible expose features/components to applications that were not intended. So valves are still there, and I suspect they will be there for quite some time.

Post new comment

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.