Jetty on a cheap virtual server

In my last blog, I wrote about my first tries on using Jetty to improve the memory footprint for using a cheap (15$) virtual private server with 300MB, running the complete application, including database and webserver. These results have been derived by using benchmarks of the main part of the application, without changing the java settings. Now it is time to come back and talk about the results after running jetty 6.1.14 in production. It should be noted that I had a 100% uptime of the application the first two month of 2009, while still running on tomcat. Shortly after switching to Jetty, the uptime dropped below 98% due to different reasons:

• Running java 1.6.0_11 in client mode was a very bad idea. Jetty simply stalled after 3hours without any error or warning message. I've seen similar behaviour with tomcat 6.0.18, while tomcat 6.0.14 has been running for weeks without problems with the same java version in client mode. I don't know why, but lesson learned: When running a server application, use the server JVM.
• A concurrency bug in my code showed up. The load didn't increase, but jetty seems to be running the code much more concurrent than tomcat did. After fixing the code, that problem was solved.
• A break-in to a different VPS on the same machine caused a lot of problems with extremly slow network and access. After the provider shut down that VPS, everything got better. This was maybe the main cause of the uptime reduction, and it has more to do with VPS handling than with Jetty.

With jetty, I managed to reduce the memory footprint. The application was still running well with new java options. On tomcat, I had to use:
JAVA_OPTS='-server -Xmx96m -Xms96m -Djava.awt.headless=true -XX:+UseConcMarkSweepGC -XX:PermSize=24m -XX:MaxPermSize=24m'
Now I'm using without problems:
JAVA_OPTS='-server -Xmx76m -Xms32m -Djava.awt.headless=true -XX:+UseSerialGC
-XX:MaxPermSize=20m -XX:ReservedCodeCacheSize=16m'
The ReserverdCodeCacheSize is a setting I just found out resently, and it is 46MB by default on a x86 server machine. With some additional tunings for the database reducing its memory requirements from 36MB to 25MB while increasing slightly the cache-hit-ratio from 1% to 2%, I'm now running the application within 220MB, and I'm not afraid of having more than 50hits/s.

Another important adjustment I made to increase the stability was to preallocate a database-connection to each running threads. Usually, a connection pool with 2-5 spare threads would be enough, but on a VPS you never know how long it takes to start a new process. I have seen a simple 'ls' waiting for 10-20s, and the 10s timeout for waiting on a database connection was often timed out, in particular in the morning when people start using my application.