TomcatExpert

Trick My Proxy: Front Apache Tomcat with HAProxy instead of Apache

posted by jbrisbin on July 12, 2010 09:14 AM

It used to be common practice to "spare" the Tomcat server the drudgery of serving static content like images, CSS stylesheets, and JavaScript because it was faster to do that with Apache. That really hasn't been the case for quite a while now. With Tomcat adopting NIO and some of the really low-level performance improvements of sendfile and Asynchronous IO (AIO), it's not strictly necessary to have Apache in front of Tomcat any more.

We're probably like a lot of folks, though, and we have to have an Apache server somewhere to support the PHP applications we have (bad idea, long story). Since we already have an Apache server, we also use it to serve our static resources, just like everyone's been doing for years. We started with mod_jk because that was the only viable option at the time. When mod_proxy got AJP support, we switched to using Apache's mod_proxy and saw a nice performance boost. We stayed with this configuration for several years just because it's not terribly complicated and it just works.

But fronting Tomcat servers with Apache is pretty limiting when you start talking private cloud architectures. I discovered some big problems when I first fired up our SpringSource tcServer instances fronted by a couple of DNS round-robin load-balanced Apache servers. Tomcat was getting confused because requests were coming through different front-end proxies. I wrote my own session manager, which I discussed recently in two blog posts (<a data-cke-saved-href=" http:="" www.tomcatexpert.com="" blog="" 2010="" 06="" 21="" clustering-cloud-friendly-tomcat-sessions-rabbitmq-part-i"="">Part I & Part II), to get around this problem. But I still had the limitation of forcing users to go to an Apache server before being able to access the dynamic resources located in my tcServer backends. This created issues when I needed to take down the server Apache was running on.

It's pretty unrealistic to think we can ever really achieve a high-availability solution that has zero single points of failure, especially if you're building your own cloud in your own data center. There's always going to be a DNS server or a proxy server or a switch that is the lynchpin of the enterprise. But it seemed to make more sense to use a more robust proxy solution as our web application entry point so we could point our users at the proxy instead of at an actual Apache server. This lets us take servers down and bring new ones up to replace them without affecting overall availability. Since I'm trying to do everything in at least twos, having a second HAProxy server with identical configuration gives me a little bit of failover.

HAProxy

HAProxy is a lightweight proxy server that advertises obscenely high throughput. Judging from the places it's being used, and the amount of TCP/IP traffic being passed through it, relying on it to handle all our production traffic didn't seem to be much of a stretch. Not only will HAProxy handle HTTP traffic, but it's a general-purpose TCP/IP proxy. We also run our RabbitMQ traffic through it to get failover and load-balancing for our messaging infrastructure. I'm even experimenting with shoving our PostgreSQL database traffic through it. This reusability appeals to my "the only uni-tasker you need is a fire extinguisher" sensibilities I talked about in a previous article.

I see my HAProxy server not as another type of application server but as a router. If I want to send web traffic bound for server A to server B while I update the packages on server A, I can tweak my HAProxy configuration and ask it to reload this configuration (and this is the deal-breaker for me not using Apache) without having to break existing connections.

Apache configuration changes result in killing hundreds of active connections. This is a killer to service availability. A human accessing your web application while Apache is restarted might experience random errors but at least they can hit "Reload" in the browser or click the link again if it doesn't do what they expect. When your services are exposed to non-human users, interrupting those requests can mean headaches for everyone involved. We have an AS/400 in our department and we do some pretty low-level integration between RPG and Java code that accesses web services provided through our tcServer application servers. RPG is not a flexible language that can handle bad input data or, quite frankly, anything other than what it expects. Service interruptions were cascading errors through this application chain which creates issues for a couple programmers and probably someone downstairs in Accounts Payable.

If one of the primary assumptions about your environment is that services like Apache or Tomcat can come up or go down at any time, then you simply have to put something in front of those services that can handle routing the traffic correctly, based on the cloud topology at the moment.

Configuration

Although we're not using sticky sessions for reasons I've previously explained, you can use HAProxy to route traffic from a particular user back to the server they were previously on. The configuration is similar to Apache mod_proxy's in that you define a text string that HAProxy will prefix to the cookie value to determine where to send that request. This works great if you now want to do DNS load-balancing and have your users hitting several different front-end servers instead of coming up with subdomains and unique server names and the like.

To configure HAProxy to do sticky sessions, you need to set up a "backend". In HAProxy, a backend is a set of servers you want to route traffic to. This would be equivalent to mod_proxy's <Proxy> directive. I have two backends configured: one for Apache and one for tcServer/Tomcat. The Apache backend is quite simple, but here's the important parts:

backend apache
 
  mode http
 
  ...
 
  option httpclose
 
  option redispatch
 
  server www1 172.23.12.24:80 check inter 2000
 
  server www2 172.23.12.25:80 check inter 2000
 
  server www3 172.23.12.26:80 check inter 200

I've omitted some of the configuration directives that relate to checking the status of the Apache servers. That's covered in the excruciatingly-detailed README.

This configuration tells HAProxy that this backend is an HTTP backend. The "httpclose" and "redispatch" options are critical when proxying HTTP servers, though I honestly couldn't tell you exactly why because the explanation given in the README was a little over my head. Suffice it to say this won't work without them.

The "server" directives tell HAProxy which backend servers to load-balance. You can configure each backend differently, but I've chosen to configure "roundrobin" load-balancing globally. You can also use a weighted balancing policy. In a dynamic environment, though, I think you'd want to generate this configuration file, with load balancer weights calculated from the specifics of the machine joining the cloud. That's way beyond the scope of this article. For most private clouds, where backend servers use known IP addresses, it would suffice to configure all of the potential backend servers. Any that are not up won't be part of the load-balancing equation, so there's no negative impact to leaving servers that are currently down configured in HAProxy. When they come back up, HAProxy will see that and start sending them traffic again. All with no restarts.

The tcServer backend configuration is also pretty simple, though this example diverges from my real configuration in that I've added the necessary cookie settings to make sticky sessions work. I have these commented out in my HAProxy configuration so I can have a backup plan in case something goes horribly wrong with RabbitMQ or I introduce a bug into my session manager. Reverting back to a sticky session configuration to restore service is as simple as uncommenting the right directives and reloading HAProxy.

backend tcserver
 
  mode http
 
  ...
 
  option httpclose
 
  option redispatch
 
  cookie JSESSIONID prefix
 
  server vm33-tc1 172.23.13.33:8180 cookie vm33tc1 check inter 5000
 
  server vm33-tc2 172.23.13.33:8280 cookie vm33tc2 check inter 5000
 
  server vm33-tc3 172.23.13.33:8380 cookie vm33tc3 check inter 5000

You'll notice we've added some references to the session cookie. In Java web applications, this should be JSESSIONID. HAProxy will do what mod_proxy does to manage sticky sessions: it will strip out part of the JSESSIONID cookie value and match it against the configured backends. If you use a tool like Firebug and look at the cookies coming back from Tomcat with this HAProxy configuration in place, you'll see something like "vm33tc1~REALLYLONGMD5HASH" for the JSESSIONID cookie value. If you don't want to use sticky sessions, delete the references to "cookie" configuration. No other changes are required.

In one of my earlier articles, I discussed hooking into JMX to monitor server lifecycle events. If you had a lifecycle listener on that queue running on the same VM as HAProxy, it could generate this list of servers and issue a soft reload. Otherwise, you can configure every server you might load balance with, disregarding their current up or down status. When they come up at some point in the future, HAProxy will see them and start load balancing against them.

We're not quite finished. We haven't yet told HAProxy what traffic to send to Apache and what traffic to send to tcServer. In mod_proxy, you'd set a ProxyPass and ProxyPassReverse directive on the path to your webapp. In HAProxy, you do something similar, but HAProxy uses "acls" (Access Control Lists).

To tell HAProxy to listen on a particular port, you need to set up a "frontend". HAproxy is a very powerful proxy server, so setting up an HTTP frontend can get pretty complicated, depending on how you're wanting to route traffic. HAProxy knows a lot about HTTP requests because it treats them differently than other TCP/IP traffic. HAProxy actually interrogates the HTTP requests and responses and can do all kinds of things to them. The full options for this are in the README, which I strongly encourage you to read.

A basic frontend for splitting up HTTP traffic bound for a tcServer application on context path "/rest" and letting all other HTTP traffic go to Apache would need at least something like:

frontend www
 
  mode http
 
  bind :80
 
  option forwardfor
 
  capture request  header ...
 
  capture response header ...
 
  acl rest path_beg /rest
 
  use_backend apache unless rest
 
  default_backend tcserver
 
There are a number of headers that need to be sent to the backend servers so Tomcat sees the client's IP address and Cache-Control headers get passed back correctly. Since what you might actually need depends on your particular setup, it's probably best you set this part up yourself. Just be aware that until you get these "capture" directives set up, your Tomcat server will think the HAProxy box is requesting the page, not the client on the other end of the proxy.

The acl configuration is pretty self-explanatory. If the request path doesn't begin with the string "/rest", send it to the Apache backend, otherwise let the tcServer backend handle it.

Monitoring

HAProxy includes a built-in HTML report on what's going on inside your proxy server. To access it, you'll need to set up a special "listen" section:

listen stats :7583
 
  mode http
 
  stats uri /

If you pull up http://haproxy1.mycloud.mycompany.com:7583/ you'll see gobs of statistics and really low-level information on what's happening. Securing this URL is covered in the fore-mentioned HAProxy README.

Fronting Tomcat with Apache is a stable and traditional configuration. It's also pretty static. If you're trying to jump on the cloud computing bandwagon and your environment just got very dynamic, or you simply want more flexibility in how you serve Tomcat-based content, consider replacing Apache and mod_proxy with the more robust HAProxy.

Jon Brisbin is an Architect/Analyst/Java Guru at NPC International, the world's largest Pizza Hut franchisee. He's been deploying web applications on Tomcat for over 10 years and currently focuses on cloud computing (virtual, hybrid, and private). He built a private cloud from scratch using VMware ESX, Ubuntu Linux, packing tape, and rusty baling wire. He's done consulting work with industry leaders and Mom-and-Pops alike. Prior to NPC, Jon developed new application frameworks, integrated the AS/400 with UNIX and Windows systems, developed Lotus Domino applications, hacked websites together with Perl CGI and a text editor, and served with US Air Force Intelligence in a very hot and sandy, but undisclosed, location. He lives in the rural Midwest.

He blogs on Web 2.0 (and sundry topics) on his website: http://jbrisbin.com/web2

Comments

Good article

That's a nice article Jon. I wasn't aware that Tomcat had improved to the point it was worth addressing it directly without passing through Apache. That's one point that people sometimes ask me when the intend to deploy haproxy, and on which I have no opinion.

There are two minor improvements that you could do on your configuration above. The first one would be to enable keep-alive on the client side. This will reduce page load time for clients over high latency networks. For this, you have to replace "option httpclose" with "option http-server-close" everywhere. You should also set "option http-pretend-keepalive" in the tomcat backend, as until very recently, it was making keep-alive impossible.

The second point is that you could enable HTTP health checks in your farms, in order to ensure that the service is really running and is not just accepting connections.

These are some great

These are some great improvements! Thanks for passing this on.

www.tomcatexpert.com - Global information

f5f6fdf8e039a087cfd771b09f83bab7 www.tomcatexpert.com Hello! http://www.tomcatexpert.com/?a2e1a254f258,

Clustered, too?

So you cluster your tomcats behind haproxy or do they run independently? When you are bringing severs on or offline, how are you handling that with haproxy?

Yes, we cluster our Tomcats.

Yes, we cluster our Tomcats. I've blogged about this recently here on tomcatexpert.com You could also use sticky sessions, which I've used successfully as well. For servers coming offline and online, there's not a terribly good way to handle that. I've found it's easier to simply pre-configure the possible servers and just let them be detected by HAProxy health checks. I'm tossing around the idea of generating the config file, which has the appropriate entries in it, but I haven't really had time to tackle that yet.

Session problem with lb'ed Apache

You wrote:
> Tomcat was getting confused because requests were coming through different front-end proxies.

Was this with sticky sessions or with replicated (cluster) sessions? Can you describe a bit more, what the problem is?

apache on red hat

guys kindly give me steps, on how to install apache in red hat..

Post new comment

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.