TomcatExpert

Crawler Session Manager Valve

posted by mthomas on May 18, 2011 07:25 AM

For organizations with large publically searchable websites, such as those found in ecommerce companies with large product catalogues or companies with active online communities, web crawlers or bots can trigger the creation of many thousands of sessions as they crawl these large sites. Normally crawling sites without relying on cookies or session IDs, these bots can create a session for each page crawled which, depending on the size of the site, may result in significant memory consumption. New in Apache Tomcat 7, a Crawler Session Manager Valve ensures that crawlers are associated with a single session - just like normal users - regardless of whether or not they provide a session token with their requests.

A Relevant Example

One of the roles I play in the Apache Tomcat project is managing the issues.apache.org servers which run the two Apache issue trackers we have—two instances of Bugzilla and one instance of JIRA. Not surprisingly, JIRA runs on Tomcat. A few months ago, while looking at the JIRA management interface, I noticed that we were seeing around 100,000 concurrent sessions. Given that there are only 60,000 registered users and less than 5,000 active users any month, this number appeared extremely inflated.

After a bit of investigation, the access logs revealed that when many of the webcrawlers (e.g., googlebot, bingbot, etc) were crawling the JIRA site, they were creating a new session for every request. For our JIRA instance, this meant that about 95% of the open sessions were left over from a bot creating a single request. For instance, a bot requesting 100 pages, would open 100 sessions. Each one of these requests would hang around in memory for about 4 hours, chewing up tremendous memory resources on the server.

The Fix

The goal for the Crawler Session Manager Valve is to ensure that when that same crawler requests those 100 pages, it only results in a single session. To do this, Tomcat uses a regular expression to see if the incoming request is from a known user agent HTTP request header (by default it checks for *[bB]ot.*|.*Yahoo! Slurp.*|.*Feedfetcher-Google.*), and it keeps a note of all the IP addresses those headers came from as well as the last Session ID of that request.

When a crawler first access the site, a new session is created as part of that first request, however upon requesting a second page – the Crawler Session Manager Valve recognizes the crawler from its user agent header, matches it to the IP address and insert the previous session ID into the request. Thus, the crawler only ever opens a single session.

Configuring the Crawler Session Manager Valve

Shipped with Tomcat 7, the Crawler Session Manager is not enabled by default. To turn on the valve, see the valve documentation at http://tomcat.apache.org/tomcat-7.0-doc/config/valve.html#Crawler_Session_Manager_Valve.

There are two main options for configuring this valve. The first is the crawlerUserAgents property which allows you to specify what bots to look for by their user agent header name. Additionally you can configure the sessionInactiveInterval which specifies how long Tomcat should hold on to the assigned session ID. It is not recommended to hold onto the session ID for more than a couple hours as these bots do tend to change their IP addresses regularly.

The Result

For the issues.apache.org site, implementing this valve on the JIRA site alone took the concurrent number of sessions average down from 100,000 to about 5,000. Additionally, there was a significant drop in resource usage on the server, and it is also now relatively simple to monitor from the Current Sessions page what web crawlers are currently active on the site and how many hits they are generating.

Special note: Although JIRA is only certified to run on Tomcat 5 and Tomcat 6, we actually run it on the latest Tomcat 7 release. Running JIRA on Tomcat 7 has not caused any issues which, as an aside, is a testament to how well Tomcat 7 and the Servlet 3.0 specification has been engineered for backwards compatibility.

Mark Thomas is a Senior Software Engineer for the SpringSource Division of VMware, Inc. (NYSE: VMW). Mark has been using and developing Tomcat for over six years. He first got involved in the development of Tomcat when he needed better control over the SSL configuration than was available at the time. After fixing that first bug, he started working his way through the remaining Tomcat bugs and is still going. Along the way Mark has become a Tomcat committer and PMC member, volunteered to be the Tomcat 4 & 7 release manager, created the Tomcat security pages, become a member of the ASF and joined the Apache Security Committee. He also helps maintain the ASF's Bugzilla instances. Mark has a MEng in Electronic and Electrical Engineering from the University of Birmingham, United Kingdom.

Comments

How do I run tomcat without any sessions?

We have a webapp that is exclusively for web service calls. There will never need to be any data retained beyond a single request (e.g., each request is completely stateless). So we really have no need to create any sessions at all for this webapp.

Is there a way to run tomcat (7.0.16) so that it doesn't waste any resources at all creating sessions?

--

Robin D. Wilson

Do we have anything similar

Do we have anything similar for apache server??

This is very appealing,

This is very appealing, however , it is very important that will mouse click on the connection: Wrap Him Around Your Finger

These you will then see the

These you will then see the most important thing, the application provides you a website a powerful important internet page: michael fiore text your ex back

I simply want to tell you

I simply want to tell you that I am new to weblog and definitely liked this blog site. Very likely I’m going to bookmark your blog . You absolutely have wonderful stories. Cheers for sharing with us your blog. The Millionaire Brain Academy

You will be permitted to

You will be permitted to posting companies, yet not one-way links, except in cases where they can be permitted plus for issue.
Cheap Club Flyers

Obat Herbal Paling Ampuh

Obat Herbal Sinusitis Tanpa Operasi Pada umumnya pasien sinusitis cuma diberi obat antibiotik, tapi sekarang sudah hadir amazon Plus yang pasti bisa menjawab solusi terpercaya bagi pasien penyakit sinusitis. obat tradisional sinusitis alami amazon Nature bisa memberi kesembuhan sampai selesai. Obat Tradisional amazon merupakan pelopor nomor satu buat obat kesehatan alami bagian Asia Tenggara, khusunya negara Indonesia.

Obat Herbal Paru Paru Basah Paling Ampuh amazon Jus Kulit Manggis diolah menggunakan bahan-bahan dari alam antara lain sari kulit manggis, bunga roselle, apel, buah anggur dan menggunakan madu sebagai bahan pengawet alami. Kandungan zat xanthone yang terdapat di dalam kulit buah manggis mempunyai super antioksidan yang bisa memulihkan segala jenis penyakit termasuk radang paru-paru.

Obat Kanker Otak Stadium 4 Tanpa Operasi terdapat senyawa molekul yang memiliki kandungan antioksidan yang dikenal zat xanthone. Dan xanthone inilah yang menginspirasi lahirnya sebuah minuman alami Amazon. Amazon jus kulit manggis dapat mengobati Kanker otak yang membuat hidup manusia menderita. Memilih Amazon jus kulit manggis merupakan pilihan bijak untuk penderita kanker otak stadium 4.

Toko Online Diherbal.com menawarkan jenis herbal berkualitas tinggi dengan tujuan untuk memperbaiki mutu kesehatan Anda. Diherbal.com merupakan toko online yang menjual obat obatan herbal bermutu.

Diherbal Online Shop

Di dalam Obat Kanker Kelenjar Getah Bening Stadium 4 Amazon Plus juga terkandung madu, berguna menutupkan juga sebagai memulihkan luka karena kelenjar getah bening, bunga rossela berguna untuk lancarnya sirkulasi darah, yang dikacaukan oleh karena pertumbuhan penyakit kanker kelenjar getah bening ini. Kita lihat kerjasama kandungan yang terdapat dalam Obat Kelenjar Getah Bening Stadium 4 Amazon Plus ini sangat menakjubkan.

Minumlah air putih segelas, sepuluh menit kemudian minum minum Amazon Plus Obat Batu Empedu Tradisional, terus minumlah air kembali, serta jangan lupa untuk berdoa. Serta janganlah berhenti minum Obat Tradisional Batu Empedu Amazon Plus hingga sembuh total. Saya minum 3 kali setiap harinya, seusai makan.

Pengobatan alternatif dengan Obat Batu Ginjal Tradisional Amazon Plus merupakan pengobatan yang sangat aman dan sudah terbukti ampuh menyembuhkan penyakit batu ginjal sampai sembuh, meskipun tidak menjalankan operasi.

Toko Obat Herbal GrosirAmazonPlus.com kami menjual segala jenis herbal berkualitas tinggi bermaksud untuk membantu meningkatkan mutu kesehatan Anda. Kami adalah pusat grosir obat herbal terbesar di Indonesia.

sl786982

Many thanks for the exciting blog posting! Simply put your blog post to my favorite blog list and will look forward for additional updates. Simply wanted to write down a word in order to say thanks to you for those wonderful tips. Clinic online

sl786982

sl786982

Chinese medicine clinic
This is my very first time that I am visiting here and I’m truly pleasurable to see everything at one place.

sl786982

No doubt this is an excellent

No doubt this is an excellent post I got a lot of knowledge after reading good luck. Theme of blog is excellent there is almost everything to read, Brilliant post.
elens

Mayline Aberdeen Series

A commitment of thankfulness is all together for your post! I recognize there are different who feel an unclear fulfillment from I read this article! I trust you will keep having such articles to concede to everybody! Mayline Aberdeen Series

This is such a great resource

This is such a great resource that you are providing and you give it away for free.
Baseball Drills

dani

Your blog provided us with valuable information to work with. Each & every tips of your post are awesome. Thanks a lot for sharing. Keep blogging,
steroids canada

Excellent .. Amazing .. I’ll

Excellent .. Amazing .. I’ll bookmark your blog and take the feeds also…I’m happy to find so many useful info here in the post, we need work out more techniques in this regard, thanks for sharing.
West Virginia Mesothelioma Law Firm

useful information on topics

useful information on topics that plenty are interested on for this wonderful post.Admiring the time and effort you put into your b!..
new futura 5 bedroom

Rent a Car Larnaca Airport

Car hire Larnaca (Rent a Car Larnaca Airport) via CosmosCarRental.com delivers a car that will take you from point A to point B, without any problems, anywhere in Cyprus. Additionally there are NO HIDDEN COSTS.

ladies nightwear

This rhetoric is evidenced within the baseball creed and has its roots in the societal upheaval caused by the rapid industrialization of America during the last half of the nineteenth century..... ladies nightwear

All the contents you

All the contents you mentioned in post is too good and can be very useful. I will keep it in mind, thanks for sharing the information keep updating, looking forward for more posts.Thanks
Plumber in Roswell, Ga

You know your projects stand

You know your projects stand out of the herd. There is something special about them. It seems to me all of them are really brilliant!
Depression

I am very enjoyed for this

I am very enjoyed for this blog. Its an informative topic. It help me very much to solve some problems. Its opportunity are so fantastic and working style so speedy.
new futura pricing

Thanks for taking the time to

Thanks for taking the time to discuss that, I feel strongly about this and so really like getting to know more on this kind of field. Do you mind updating your blog post with additional insight? It should be really useful for all of us.
new futura price list

dani

I like this post,And I guess that they having fun to read this post,they shall take a good site to make a information,thanks for sharing it to me.
new futura developer

This type of message always

This type of message always inspiring and I prefer to read quality content, so happy to find good place to many here in the post, the writing is just great, thanks for the post.
Native Ads

Thanks for sharing this

Thanks for sharing this information. I really like your blog post very much. You have really shared a informative and interesting blog post with people..
https://gilbertway.net

I am happy to find this post

I am happy to find this post very useful for me, as it contains lot of information. I always prefer to read the quality content and this thing I found in you post. Thanks for sharing.
click this site

I was looking at some of your

I was looking at some of your posts on this website and I conceive this web site is really instructive! Keep putting up..
детско спално бельо

dani

I’ve been searching for some decent stuff on the subject and haven't had any luck up until this point, You just got a new biggest fan!..
look at here

dani

Thanks for this great post, i find it very interesting and very well thought out and put together. I look forward to reading yohow to get my walmart paystub
ur work in the future.

Nice knowledge gaining

Nice knowledge gaining article. This post is really the best on this valuable topic.
pilates clontarf

artemisinin.rs

It's really nice and meanful. it's really cool blog. Linking is very useful thing.you have really helped lots of people who visit blog and provide them usefull information. artemisinin.rs

Thanks for the nice blog. It

Thanks for the nice blog. It was very useful for me. I'm happy I found this blog. Thank you for sharing with us,I too always learn something new from your post.
Children’s Speech Pathologist Adelaide

DANi

Thank you for some other informative blog. Where else could I get that type of information written in such an ideal means? I have a mission that I’m just now working on, and I have been at the look out for such information.
My Lowe's Life

Interesting and amazing how

Interesting and amazing how your post is! It Is Useful and helpful for me That I like it very much, and I am looking forward to Hearing from your next..
Discount Sale

I would like to thank you for

I would like to thank you for the efforts you have made in writing this article. I am hoping the same best work from you in the future as well..
Mesothelioma Cancer

I was surfing net and

I was surfing net and fortunately came across this site and found very interesting stuff here. Its really fun to read. I enjoyed a lot. Thanks for sharing this wonderful information.
testogen customer reviews

DANI

Thank you for another great article. Where else could anyone get that kind of information in such a perfect way of writing? I have a presentation next week, and I am on the look for such information.
Lelio Vieira Carneiro Junior

online io games list

Solid content. Keep up the amazing work I can't wait to read more on this topic from you. online io games list

DANI

I really like your writing style, great information, thankyou for posting.
click this site

Thanks for posting this info.

Thanks for posting this info. I just want to let you know that I just check out your site and I find it very interesting and informative. I can't wait to read lots of your posts.
click this site

multiplayer games

Very informative post ! There is a lot of information here that can help any business get started with a successful social networking campaign ! multiplayer games

DANI

nice bLog! its interesting. thank you for sharing....
click this site

ZDSWR

Thanks for sharing the info, keep up the good work going.... I really enjoyed exploring your site. good resource...
Marrakech Trip

Really a great addition. I

Really a great addition. I have read this marvelous post. Thanks for sharing information about it. I really like that. Thanks so lot for your convene.
swifty buys houses

Thanks for every other

Thanks for every other informative site. The place else may just I get that kind of information written in such an ideal means? I have a venture that I’m just now operating on, and I have been on the look out for such information.
Dolce Vita

Sa

Your website is really cool and this is a great inspiring article.
stempel

This is such a great resource

This is such a great resource that you are providing and you give it away for free. I love seeing websites that understand the value of providing a quality resource for free. It is the old what goes around comes around routine.
Industrieller

Sa

Great articles and great layout. Your blog post deserves all of the positive feedback it’s been getting.
Tourist Attractions

Sa

Good post but I was wondering if you could write a litter more on this subject? I’d be very thankful if you could elaborate a little bit further. Appreciate it!
Tourist Attractions

Play Run 3

Very informative post ! There is a lot of information here that can help any business get started with a successful social networking campaign ! Play Run 3

Post new comment

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.