TomcatExpert

Explaining jdbc-pool, a High Concurrency Alternative to Connection Pooling Module

posted by fhanik on March 12, 2010 02:23 PM

Why another connection pool project

In this article we are going to explore one of the modules that is being developed in the Tomcat development trunk. The module named, jdbc-pool, is exactly that, an implementation of pooling of JDBC resources.

Connection pools have been around for a long time. I wrote one myself in 1997 when I first was introduced to Java, and used that for a long time in different projects I worked on. When JDBC came up, developers quickly realized that pooling JDBC connections would be beneficial for performance, and also a way to siphon connections and control the resource usage on the database server. This was especially important in horizontally scaled clusters where the database was always a risk of being overloaded. In those days we typically ran 1, 2 and sometimes 4 CPU machines for the application servers.

Connection pooling quickly became something that J2EE vendors developed in-house to integrate with their transaction services and became proprietary and integrated into the server itself. Outside of the J2EE containers a number of open source projects popped up. A few examples are Proxool, Apache Commons DBCP, C3P0 and Poolman. These are the most common ones we encounter in the field. Apache Tomcat has for very many years relied on DBCP to provide Tomcat users with a connection pooling implementation.

All of these open source projects have their pros and cons, but they all have something in common. They have become fairly stagnant and the code bases have not continued to evolve to meet new features in the Java language and are not optimized for multicore hardware that is increasingly becoming the common platform to deploy Java applications on. Some of the projects have also suffered from unneeded complexity in the source code, most likely one of the main reasons the project code is not moving forward, making it hard to maintain and easy to break.

Out of the four open source projects mentioned above, the two we see in Apache Tomcat deployments today are Commons DBCP and C3P0. A short summary of these two projects would be:

Commons DBCP

  • Ships with Apache Tomcat, used because it is readily available
  • Fairly stable code base
  • Plagued by dead locks for a very long time
  • Customer on lower clock speed but higher parallelism often complain about throughput bottlenecks

C3P0

  • Often used instead of DBCP
  • Based on our experience in the field, still fairly buggy
  • Reputation to perform better than DBCP, albeit our results show different

Both of these implementations are fairly complex. DBCP is built on top of the Apache Commons Pool project, and all together forms a code base of over 60 source files. C3P0 has grown to over 200 source files. Yet both pools fall short on features, to mention a few:

  • Pools are often statically tied to a Java version, meaning if the pool was written for Java 5, it wont have the java.sql and javax.sql interfaces compatible with Java 6. A user would typically receive a NoSuchMethodException when trying to execute one of the newer interface methods.
  • The pools are limited to java.sql.Driver interface, making it impossible to pool resources from a javax.sql.DataSource
  • No support for XA data sources
  • No flexibility for plugging in new features

So back to our question, why yet another pool project? The answer was simple, we needed better performance and more features. We also believed that taking on one of the legacy implementations would require rewriting a majority of the code base. So, it came to make a judgement call, do we want to rewrite 60+ sources files, or even 200+.

I know what you're probably thinking, here is yet another engineer that believes in rewriting from scratch rather than contributing to the existing code base. In almost every case, I prefer to reuse and fix than rewrite. However, after carefully judging the following questions:

  • How many lines of code do we actually need to provide the exact same implementation of Commons DBCP?
  • Can we make my implementation flexible enough to expose all the methods of java.sql.Connection that the driver exposes?
  • Can we provide a framework for exposing additional features?
  • Can we make the implementation scale in multi core environments/?

The next challenge we faced with such a project, within the Apache Software Foundation, was how to build a community around the project. Community is often grown by having lots of code that is not quite finished, meaning, there is work to be done in multiple areas making it enjoyable for a group to work together.

By this time, we had a prototype, it was 8 source files, and provided all the features of the other pools combined. Not all that exciting for a group of Java developers. Trying to fire off a separate project would probably not be very fruitful from a community perspective. We've since then grown it to 24 source files mostly by creating prebuilt addons that can be configured.

And so the jdbc-pool module was started.

The jdbc-pool did get some early adoption and the number of bugs was very low. Most early bugs were actual workarounds for typical use of the java.sql interfaces, calling close multiple times on a connection etc.

Today we see jdbc-pool, even in its unreleased state, several cases of production use. SpringSource ships the pool module with its tc Server product and you can already read about it in blog posts.

The jdbc-pool has an almost identical configuration as the Commons DBCP pool, in order to provide an easy migration path. For users already using Apache Tomcat and DBCP, the only difference would be the factory attribute  factory="org.apache.tomcat.jdbc.pool.DataSourceFactory".

For users of dependency injection frameworks, like the Spring Framework it's easy to just configure the org.apache.tomcat.jdbc.pool.DataSource bean.

Folks, that is why we have yet another pool module for JDBC connections. Over the next few articles, I will be digging deeper into:

  • Performance of jdbc-pool
  • The feature set, how and when to use it
  • The actual implementation and why it scales
  • How to build extensions

Filip Hanik is a Senior Software Engineer for the SpringSource Division of VMware, Inc. (NYSE: VMW) and a key participant in the company's Apache Tomcat initiatives. Filip brings 15 years of extensive experience in architecture, design and development of distributed application frameworks and containers and is recognized for his top-quality system development skills and continuous participation of Open Source development projects. Filip is an Apache Software Foundation member and a committer to the Apache Tomcat project where he is a leading authority on Tomcat clustering and a key contributor to the core of the platform. Filip has made contributions to software initiatives for Walmart.com, Sony Music, France Telecom and has held a variety of senior software engineering positions with technology companies in both the United States and Sweden. He received his education at Chalmers University of Technology in Gothenburg, Sweden where he majored in Computer Science and Computer Engineering.

Comments

Switching from C3P0...

Hi Filip.


I'm using C3P0 in a product I work on. I occasionally have problems with deadlocks, but in general it works pretty well. Do you think jdbc-pool is mature enough to start considering an eventual switch?



--Matt

RE: Switching from C3P0...

Hi Matt, absolutely. The jdbc-pool has faired very well in many environments that we've seen. So go for the switch.
What we see as common pitfalls, is that some code doesn't have proper closure of statements, for that they would use the StatementFinalizer, or applications don't reset auto commit and read only values, and for that they would use the ConnectionState interceptor.
We have an upcoming article about configuration, that will talk more in depth about these issues.

If you run into any problems, you can ping me on my apache.org email, my id is fhanik, or ask a question on the tomcat user mailing list.

best
Filip

RE: Switching from C3P0...

Thanks Filip. I'm working on a product that's about to have a release. Once that's out, I'll give jdbc-pool a try. Maybe I'll be able to include it in the next release. :)

--Matt

Sorry for the slow reply - for some reason my dev build of Chrome is consistently unhappy with the reply page. It took me a few days to get around to writing it in Firefox. ;)

On this page, you'll see my

On this page, you'll see my profile, please read this information. Save The Marriage System

On this page, you'll see my

On this page, you'll see my profile, please read this information. Ways To Get Your Ex Back

Welcome to the party of my

Welcome to the party of my life here you will learn everything about me. Survive In Bed

This is helpful, nonetheless

This is helpful, nonetheless it can be crucial so that you can check out the following website: audiometry

This is very appealing,

This is very appealing, however , it is very important that will mouse click on the connection: How To Give A Hand Job

During this website, you will

During this website, you will see this shape, i highly recommend you learn this review. f4x bodybuilding rebelmouse

You possess lifted an

You possess lifted an essential offspring..Blesss for using..I would want to study better latest transactions from this blog..preserve posting.. How to Turn a Guy On

There you can download for

There you can download for free, see the first of these data. sexosur

Very interesting information,

Very interesting information, worth recommending. However, I recommend this: sexosur

Mmm.. estimable to be here in

Mmm.. estimable to be here in your report or notify, whatever, I repute I should moreover process strong for my have website want I play some salubrious further updated busy in your location. sexonorte

This is important, though

This is important, though it's necessary to help you head over to it weblink: zonaescort.cl

On this page you can read

On this page you can read my interests, write something special. The Bonding Code Review

great post

hello all, introduce me from Indonesia.
I manage the best quality wholesale furniture business, for those of you who are in need of furniture can directly contact me. http://www.karyafurniturejepara.com

mebel murah mebel jati furniture online kursi tamu

Good dear!

Blessings come in many disguises. There's always something to learn from everything we go through. http://goatripsindia.com/goa-new-year-packages

Beautifully written article.

Beautifully written article. This is exactly why I've bookmarked your blog. I always want to read your articles, new or old. I hope you will keep this going on.
Ink Cartridges

sl786982

I believe your site is precisely exactly what I've been searching for; this solutions a lot of associated with my personal queries. michael spencer gilroy

sl786982

Thanks for publishing this

Thanks for publishing this amazing post today. You've been doing a really good job for the past few months and check interesting article. I just hope that you will keep doing the same job!!

I had been having some

I had been having some problems on this topic, but after reading this tremendously written blog post, all my queries have vanished. Good job! learn how to save energy

i read a lot of stuff and i

i read a lot of stuff and i found that the way of writing to clearifing that exactly want to say was very good so i am impressed and ilike to come again in future..
Seaside Residences Frasers

I found that site very

I found that site very usefull and this survey is very cirious, I ' ve never seen a blog that demand a survey for this actions, very curious...
how much does an electrician make

Nice to read your article! I

Nice to read your article! I am looking forward to sharing your adventures and experiences.
go at this site

The information I have gotten

The information I have gotten from this articles is absolutely essential for me. I am very thankful to the writer who has shared all of this information with us. Keep giving more articles like this. When Is Christmas

Good post but I was wondering

Good post but I was wondering if you could write a litter more on this subject? I’d be very thankful if you could elaborate a little bit further. Appreciate it!
des moines tattoo

I really impressed after read

I really impressed after read this because of some quality work and informative thoughts . I just wanna say thanks for the writer and wish you all the best for coming!.
carry on sizes

Great content material and

Great content material and great layout. Your website deserves all of the positive feedback it’s been getting.
typseri.top

Most of the time I don’t make

Most of the time I don’t make comments on websites, but I'd like to say that this article really forced me to do so. Really nice post!
where to buy spartagen xt

I like this post,And I guess

I like this post,And I guess that they having fun to read this post,they shall take a good site to make a information,thanks for sharing it to me.
nyc flower delivery

I had an amazing time reading

I had an amazing time reading this post. This has to be one of the best posts that I've read in recent times. I am really thankful to you for giving us this. I just hope you'll keep this going on. Halaman utama

jams

Its a great pleasure reading your post.Its full of information I am looking for and I love to post a comment that "The content of your post is awesome" Great work.
rear extension architect

I am happy to find this post

I am happy to find this post very useful for me, as it contains lot of information. I always prefer to read the quality content and this thing I found in you post. Thanks for sharing.
Stanford Pelage Blog

sl786982

This will be to an incredible degree noteworthy for me when I get an opportunity to begin my blog. Photo débutant

sl786982

Your articles are always a

Your articles are always a joy to read. The information that I collected from this article was very helpful for me to complete my project on this topic. I just hope that you will continue to give more stuff like this. débarras maison gratuit

I really loved reading your

I really loved reading your blog. It was very well authored and easy to understand. Unlike other blogs I have read which are really not that good.Thanks alot!
kyle

Wow, cool post. I’d like to

Wow, cool post. I’d like to write like this too – taking time and real hard work to make a great article… but I put things off too much and never seem to get started. Thanks though.
versicherungdrohne.de

sl786982

First of all, i would like to congratulate all the graduate in business administration. I have read this post and came to know about the idea of students in tourism business. It is really an appreciable idea. Thank you for sharing these information. Real Estate Merewether Heights

sl786982

sl786982

Thanks for your information, it was really very helpfull. newcastle real estate

sl786982

Thank you very much for this

Thank you very much for this useful article. I like it.
customer service in spanish translation

sl786982

I truly thank you for the important information on this extraordinary subject and anticipate more awesome posts. Ver web

sl786982

sl786982

You have made an awesome showing on this article. It's exceptionally lucid and exceedingly astute. Mas info

sl786982

This is actually the kind of

This is actually the kind of information I have been trying to find. Thank you for writing this information.
make money online

I love the way you write and

I love the way you write and share your niche! Very interesting and different! Keep it coming!
price for norditropin

I would like to say that this

I would like to say that this blog really convinced me to do it! Thanks, very good post. minecraft pocket edition gratuit /
free netflix accounts

Nice post! This is a very

Nice post! This is a very nice blog that I will definitively come back to more times this year! Thanks for informative post.
vobizen wise 5 booking

This type of message always

This type of message always inspiring and I prefer to read quality content, so happy to find good place to many here in the post, the writing is just great, thanks for the post.
diabetesdestroyer.strikingly.com

Thanks for posting this info.

Thanks for posting this info. I just want to let you know that I just check out your site and I find it very interesting and informative. I can't wait to read lots of your posts.
duckness.net – Best Home Interior and Decoration Ideas

I have not any word to

I have not any word to appreciate this post.....Really i am impressed from this post....the person who create this post it was a great human..thanks for shared this with us.
redmi note 4 flipkart

I was looking at some of your

I was looking at some of your posts on this website and I conceive this web site is really instructive! Keep putting up..
Canadian online pharmaceuticals

Post new comment

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.