posted by avanabs on March 19, 2010 06:34 AM

How to lose a half million dollars on one bug!

I want to share with you a recent experience from one of my clients. They have been using Apache Tomcat for several years, often combined with WebSphere (they are a “Big Blue” shop end to end) for supporting back office stateful applications. Early on, they decided to support Tomcat themselves, primarily because they did not find any viable vendors, but mostly because the development team (who had been using Tomcat on their desktops) convinced management that this would be “virtually free”. “No problem!

This has worked very well, particularly while the original Tomcat aficionado continued to provide "support" and...unknown to management...enhancements. There have been only occasional issues, easily handled by their in-house application programmers, with occasional help from the Tomcat community. No one even bothered to keep track of time spent maintaining Tomcat, because it was “free”. Over that time, however, their Tomcat version has diverged from the Apache project, because maintaining compatibility wasn't an objective, because the cost (mostly developer time) to submit their fixes to the Apache community wasn’t in anyone budget, and partly because re-integrating ongoing Apache changes was also un-funded drudgery. So, this organization is “using Tomcat” as far as management knows, but is actually using a diverging branch. All that said, the process continued to work fine and the visible costs were indeed low.

About a month ago, a new application was developed and put into test. This application was fairly simple, but it was projected to generate $ 100,000/week (TINY by their standards) initially, ramping up to over a half million/week by Q3. It was also the first visible peek at a new business strategy. The problem was that the application failed erratically during test. Subsequent debugging indicated this was due to a bug in Tomcat, not the application, so one of their application guru’s quickly rolled out a Tomcat bug fix (enhancement???) and delivered the result back to test.

At first, everything seemed fine. The new application worked great and passed thru QA with flying colors. “Free” self support won again…or did it? Application developers working on other projects fairly rapidly found that the new version was breaking some of their legacy applications, including several commercial apps. So, the application developer went back to the drawing board and quickly generated another fix.The new application failed, as did almost everything they tried to run. So, the process continued, with more than two dozen fixes generated and tried, and generated and tried, and…

Read More