TomcatExpert

Crawler Session Manager Valve

posted by mthomas on May 18, 2011 07:25 AM

For organizations with large publically searchable websites, such as those found in ecommerce companies with large product catalogues or companies with active online communities, web crawlers or bots can trigger the creation of many thousands of sessions as they crawl these large sites. Normally crawling sites without relying on cookies or session IDs, these bots can create a session for each page crawled which, depending on the size of the site, may result in significant memory consumption. New in Apache Tomcat 7, a Crawler Session Manager Valve ensures that crawlers are associated with a single session - just like normal users - regardless of whether or not they provide a session token with their requests.

A Relevant Example

One of the roles I play in the Apache Tomcat project is managing the issues.apache.org servers which run the two Apache issue trackers we have—two instances of Bugzilla and one instance of JIRA. Not surprisingly, JIRA runs on Tomcat. A few months ago, while looking at the JIRA management interface, I noticed that we were seeing around 100,000 concurrent sessions. Given that there are only 60,000 registered users and less than 5,000 active users any month, this number appeared extremely inflated.

After a bit of investigation, the access logs revealed that when many of the webcrawlers (e.g., googlebot, bingbot, etc) were crawling the JIRA site, they were creating a new session for every request. For our JIRA instance, this meant that about 95% of the open sessions were left over from a bot creating a single request. For instance, a bot requesting 100 pages, would open 100 sessions. Each one of these requests would hang around in memory for about 4 hours, chewing up tremendous memory resources on the server.

The Fix

The goal for the Crawler Session Manager Valve is to ensure that when that same crawler requests those 100 pages, it only results in a single session. To do this, Tomcat uses a regular expression to see if the incoming request is from a known user agent HTTP request header (by default it checks for *[bB]ot.*|.*Yahoo! Slurp.*|.*Feedfetcher-Google.*), and it keeps a note of all the IP addresses those headers came from as well as the last Session ID of that request.

When a crawler first access the site, a new session is created as part of that first request, however upon requesting a second page – the Crawler Session Manager Valve recognizes the crawler from its user agent header, matches it to the IP address and insert the previous session ID into the request. Thus, the crawler only ever opens a single session.

Configuring the Crawler Session Manager Valve

Shipped with Tomcat 7, the Crawler Session Manager is not enabled by default. To turn on the valve, see the valve documentation at http://tomcat.apache.org/tomcat-7.0-doc/config/valve.html#Crawler_Session_Manager_Valve.

There are two main options for configuring this valve. The first is the crawlerUserAgents property which allows you to specify what bots to look for by their user agent header name. Additionally you can configure the sessionInactiveInterval which specifies how long Tomcat should hold on to the assigned session ID. It is not recommended to hold onto the session ID for more than a couple hours as these bots do tend to change their IP addresses regularly.

The Result

For the issues.apache.org site, implementing this valve on the JIRA site alone took the concurrent number of sessions average down from 100,000 to about 5,000. Additionally, there was a significant drop in resource usage on the server, and it is also now relatively simple to monitor from the Current Sessions page what web crawlers are currently active on the site and how many hits they are generating.

Special note: Although JIRA is only certified to run on Tomcat 5 and Tomcat 6, we actually run it on the latest Tomcat 7 release. Running JIRA on Tomcat 7 has not caused any issues which, as an aside, is a testament to how well Tomcat 7 and the Servlet 3.0 specification has been engineered for backwards compatibility.

Mark Thomas is a Senior Software Engineer for the SpringSource Division of VMware, Inc. (NYSE: VMW). Mark has been using and developing Tomcat for over six years. He first got involved in the development of Tomcat when he needed better control over the SSL configuration than was available at the time. After fixing that first bug, he started working his way through the remaining Tomcat bugs and is still going. Along the way Mark has become a Tomcat committer and PMC member, volunteered to be the Tomcat 4 & 7 release manager, created the Tomcat security pages, become a member of the ASF and joined the Apache Security Committee. He also helps maintain the ASF's Bugzilla instances. Mark has a MEng in Electronic and Electrical Engineering from the University of Birmingham, United Kingdom.

Comments

How do I run tomcat without any sessions?

We have a webapp that is exclusively for web service calls. There will never need to be any data retained beyond a single request (e.g., each request is completely stateless). So we really have no need to create any sessions at all for this webapp.

Is there a way to run tomcat (7.0.16) so that it doesn't waste any resources at all creating sessions?

--

Robin D. Wilson

Do we have anything similar

Do we have anything similar for apache server??

This is very appealing,

This is very appealing, however , it is very important that will mouse click on the connection: Wrap Him Around Your Finger

These you will then see the

These you will then see the most important thing, the application provides you a website a powerful important internet page: michael fiore text your ex back

I simply want to tell you

I simply want to tell you that I am new to weblog and definitely liked this blog site. Very likely I’m going to bookmark your blog . You absolutely have wonderful stories. Cheers for sharing with us your blog. The Millionaire Brain Academy

You will be permitted to

You will be permitted to posting companies, yet not one-way links, except in cases where they can be permitted plus for issue.
Cheap Club Flyers

Obat Herbal Paling Ampuh

Obat Herbal Sinusitis Tanpa Operasi Pada umumnya pasien sinusitis cuma diberi obat antibiotik, tapi sekarang sudah hadir amazon Plus yang pasti bisa menjawab solusi terpercaya bagi pasien penyakit sinusitis. obat tradisional sinusitis alami amazon Nature bisa memberi kesembuhan sampai selesai. Obat Tradisional amazon merupakan pelopor nomor satu buat obat kesehatan alami bagian Asia Tenggara, khusunya negara Indonesia.

Obat Herbal Paru Paru Basah Paling Ampuh amazon Jus Kulit Manggis diolah menggunakan bahan-bahan dari alam antara lain sari kulit manggis, bunga roselle, apel, buah anggur dan menggunakan madu sebagai bahan pengawet alami. Kandungan zat xanthone yang terdapat di dalam kulit buah manggis mempunyai super antioksidan yang bisa memulihkan segala jenis penyakit termasuk radang paru-paru.

Obat Kanker Otak Stadium 4 Tanpa Operasi terdapat senyawa molekul yang memiliki kandungan antioksidan yang dikenal zat xanthone. Dan xanthone inilah yang menginspirasi lahirnya sebuah minuman alami Amazon. Amazon jus kulit manggis dapat mengobati Kanker otak yang membuat hidup manusia menderita. Memilih Amazon jus kulit manggis merupakan pilihan bijak untuk penderita kanker otak stadium 4.

Toko Online Diherbal.com menawarkan jenis herbal berkualitas tinggi dengan tujuan untuk memperbaiki mutu kesehatan Anda. Diherbal.com merupakan toko online yang menjual obat obatan herbal bermutu.

Diherbal Online Shop

Di dalam Obat Kanker Kelenjar Getah Bening Stadium 4 Amazon Plus juga terkandung madu, berguna menutupkan juga sebagai memulihkan luka karena kelenjar getah bening, bunga rossela berguna untuk lancarnya sirkulasi darah, yang dikacaukan oleh karena pertumbuhan penyakit kanker kelenjar getah bening ini. Kita lihat kerjasama kandungan yang terdapat dalam Obat Kelenjar Getah Bening Stadium 4 Amazon Plus ini sangat menakjubkan.

Minumlah air putih segelas, sepuluh menit kemudian minum minum Amazon Plus Obat Batu Empedu Tradisional, terus minumlah air kembali, serta jangan lupa untuk berdoa. Serta janganlah berhenti minum Obat Tradisional Batu Empedu Amazon Plus hingga sembuh total. Saya minum 3 kali setiap harinya, seusai makan.

Pengobatan alternatif dengan Obat Batu Ginjal Tradisional Amazon Plus merupakan pengobatan yang sangat aman dan sudah terbukti ampuh menyembuhkan penyakit batu ginjal sampai sembuh, meskipun tidak menjalankan operasi.

Toko Obat Herbal GrosirAmazonPlus.com kami menjual segala jenis herbal berkualitas tinggi bermaksud untuk membantu meningkatkan mutu kesehatan Anda. Kami adalah pusat grosir obat herbal terbesar di Indonesia.

sl786982

Many thanks for the exciting blog posting! Simply put your blog post to my favorite blog list and will look forward for additional updates. Simply wanted to write down a word in order to say thanks to you for those wonderful tips. Clinic online

sl786982

sl786982

Chinese medicine clinic
This is my very first time that I am visiting here and I’m truly pleasurable to see everything at one place.

sl786982

No doubt this is an excellent

No doubt this is an excellent post I got a lot of knowledge after reading good luck. Theme of blog is excellent there is almost everything to read, Brilliant post.
elens

Mayline Aberdeen Series

A commitment of thankfulness is all together for your post! I recognize there are different who feel an unclear fulfillment from I read this article! I trust you will keep having such articles to concede to everybody! Mayline Aberdeen Series

This is such a great resource

This is such a great resource that you are providing and you give it away for free.
Baseball Drills

dani

Your blog provided us with valuable information to work with. Each & every tips of your post are awesome. Thanks a lot for sharing. Keep blogging,
steroids canada

Excellent .. Amazing .. I’ll

Excellent .. Amazing .. I’ll bookmark your blog and take the feeds also…I’m happy to find so many useful info here in the post, we need work out more techniques in this regard, thanks for sharing.
West Virginia Mesothelioma Law Firm

useful information on topics

useful information on topics that plenty are interested on for this wonderful post.Admiring the time and effort you put into your b!..
new futura 5 bedroom

Rent a Car Larnaca Airport

Car hire Larnaca (Rent a Car Larnaca Airport) via CosmosCarRental.com delivers a car that will take you from point A to point B, without any problems, anywhere in Cyprus. Additionally there are NO HIDDEN COSTS.

ladies nightwear

This rhetoric is evidenced within the baseball creed and has its roots in the societal upheaval caused by the rapid industrialization of America during the last half of the nineteenth century..... ladies nightwear

All the contents you

All the contents you mentioned in post is too good and can be very useful. I will keep it in mind, thanks for sharing the information keep updating, looking forward for more posts.Thanks
Plumber in Roswell, Ga

You know your projects stand

You know your projects stand out of the herd. There is something special about them. It seems to me all of them are really brilliant!
Depression

I am very enjoyed for this

I am very enjoyed for this blog. Its an informative topic. It help me very much to solve some problems. Its opportunity are so fantastic and working style so speedy.
new futura pricing

Thanks for taking the time to

Thanks for taking the time to discuss that, I feel strongly about this and so really like getting to know more on this kind of field. Do you mind updating your blog post with additional insight? It should be really useful for all of us.
new futura price list

dani

I like this post,And I guess that they having fun to read this post,they shall take a good site to make a information,thanks for sharing it to me.
new futura developer

Make the most of mainly

Make the most of mainly premium substances - you will find him or her for: Law of Devotion

Regular visits listed here

Regular visits listed here are the easiest method to appreciate your energy, which is why why I am going to the website everyday, searching for new, interesting info. Many, thank you! ipad screen repair houston

The most interesting text on

The most interesting text on this interesting topic that can be found on the net ... Obsession Phrases

It has fully emerged to crown

It has fully emerged to crown Singapore's southern shores and undoubtedly placed her on the global map of residential landmarks. I still scored the more points than I ever have in a season for GS. I think you would be hard pressed to find somebody with the same consistency I have had over the years so I am happy with that. hampton bay fans catalog

I think this is an

I think this is an informative post and it is very useful and knowledgeable. therefore, I would like to thank you for the efforts you have made in writing this article. iSEO Company

This is very significant, and

This is very significant, and yet necessary towards just click this unique backlink: seguridad empresas

I invite you to the page

I invite you to the page where see how much we have in common. Print Signage

I should assert barely that

I should assert barely that its astounding! The blog is informational also always fabricate amazing entitys. Love Traction Lines

This is the most amazing

This is the most amazing article that I've read on the internet in a long while. This truly is top notch work from you, mate. You've shown here once again why you are the best. Please, keep it going!! Copier Drum

This is one of the most

This is one of the most fascinating articles that I've read online in the past few months. I want to thank the writer for writing this. I hope he can do more stuff like this. débarras gratuit paris

Gamerbolt.com

One minor complaint is some instances where characters repeated their lines on a multitude of occasions, most notably in boss fights. That being said, this diminishes less than a fraction of the experience. Gamerbolt.com

online dog trainer course

Hello there. This is a to a great degree elegantly composed article. I'll make sure to bookmark it and come back to peruse a greater amount of your valuable data. A debt of gratitude is in order for the post. I'll unquestionably rebound. online dog trainer course

artemisinin.rs

Hi. This is a to an incredible degree richly made article. I'll make a point to bookmark it and return to scrutinize a more noteworthy measure of your significant information.artemisinin.rs

ip booter

webstresser.org is the strongest IP Stresser / Booter on the market, we provide strongest and most reliable server stress testing ip booter

Get offers from doctors

This is additionally a decent post which I truly appreciated perusing. It is few out of every odd day that I have the likelihood to see something like this..Get offers from doctors

I simply want to tell you

I simply want to tell you that I am new to weblog and definitely liked your weblog funny dogs. Very likely I’m going to save your weblog . You will have awesome encounters. Regards for talking about with us your weblog.

thanks for j sharing with

thanks for j sharing with that awesome articles thanks u a lot
rasoio elettrico | regolabarba | ferro da stiro con caldaia | Friggitrice ad Aria | scopa a vapore | ripetitore wifi | ripetitore wifi amplificatore wifi piastra a vapore | miglior aspirapolvere | skateboard elettrico | passeggino leggero | miglior smartphone | tablet per bambini tnp)in

Mayline Napoli Series

Aw, this was a to a great degree great post. Investing some energy and real push to make a decent article… yet what would i be able to state…Mayline Napoli Series

webapp

We have a webapp that is for web assistance telephone cellphone calls. There will never need to be details managed beyond only one requirement (e.g., each requirement is very stateless). So we really have no need to create any sessions at all for this webapp see page.

sl786982

Your way to enlighten everything on this blog is actually pleasant, everyone manage to efficiently be familiar with it, Thanks a great deal. olimpiadas 2016

sl786982

ip booter

our hand crafted source, to our specially crafted DDoS apparatus we promise you will appreciate novel quality we are putting forth. ip booter

www.restascouts.com

This causes clients to locate the correct eatery, bistro and bar. At restascouts you can discover audits on eateries and bars. Compose a survey of your most loved eatery and enable different clients to locate a decent eatery. www.restascouts.com

whohammer.com

Magnificent to set off to your blog yet again, it has been quite a while for me. Well this article ive been sat tight for hence long. I require this article to complete my endeavor inside the workforce, and it has same point together with your article. whohammer.com

Truly, this article is really

Truly, this article is really one of the very best in the history of articles. I am a antique ’Article’ collector and I sometimes read some new articles if I find them interesting. And I found this one pretty fascinating and it should go into my collection. Very good work! bodrum bayan escort

GAME HACK

8 ball pool I exploit solely premium quality products -- you will observe these individuals on: coc hack

download software mxf

download software mxf converter, convert p2 mxf files from your camcorder. convert mxf files to avi mp4 mov 09 s

Everyone loves it whenever

Everyone loves it whenever people get together and share ideas. Great site, keep it up!
Mortgage Broker Calgary

Post new comment

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.