Testing hsqldb database for the cloud

Cloud services have been around now for some time. While my current preferred deployment architecture are virtual servers (VPS), those VPS have not seen much attention lately.  My current VPS costs still 15$/month, the same as in 2005, though it now has 512MB of RAM rather than 64MB and 3x the disk-space. A comparable server in the cloud should be much cheaper (approx. 0$/month). But before switching to a PaaS like Google AppEnginge or IaaS like Amazon EC2, some technical issues need to be overcome, in particular, none of these offer access to the filesystem or SQL databases for free. Google is just started to solve the reprogramming issue with Google Cloud SQL, but this is not available for free.

My application Heinzelnisse comes with 5 different information databases, 3 of them are static, or at least upgraded less frequently than the application itself, while the remaining 2 are a forum and a wiki. The idea is now to move the static dbs from the MySQL database to the application war/jar file. The 3 static dbs come from spreadsheet tables and are in spreadsheet format less than 5MB in total. Therefore, I'm trying to redeploying them to an embedded database which can be run from a jar.

I used hsqldb in the current version: 2.2.5. Smaller changes to the SQL-schema was needed to create a database. I had to make sure to use the removeAbandoned=true for the DBCP connection pool settings.

When using the default in-memory tables table-data increased from 1MB in the spreadshead table to 30MB in the data.script startup file. The performance was comparable to MySQL, I still could serve about 30 requests/s as long as I used the server jvm. The problem with in memory tables is the long startup and huge memory requirements. It took 184s to read the .script file on startup. While the application was running stable with -Xmx76MB before, I had to increase this now to more than 160MB. Using that amount of memory for a 1MB file was not acceptable.

Then I change the in-memory tables to disk-cached tables. The startup time decreased to 12s again, and application was stable again with 76MB jvm memory. But the data is now stored on disk in a .data file, which is 48MB in my case. Performance didn't change, at least not in my test with no change on the query and therefore perfect caching possibilities. So far, so nice, but file-access is not allowed in the cloud. hsqldb allows resource tables in a jar, but it wasn't clear if they work with disk-cached tables.

Testing disk-cached tables from a jar didn't work, so I had to ask in the mailing list if it is supported at all. The answer was fast and positive, even with some tips to do so, but I still didn't get it working. After scanning through the code a found a obvious bug, and my patch is accepted for the coming 2.2.6, but it is obvious that this type of deployment has not been used since the start of the 1.9 release some years ago.

I tried then also with the 1.8 release, which is still the most stable version of hsqldb. This required again minor changes in the schema, but I got it installed fast. Unfortunately,  performance of 1.8 seems to be much worse than in 2.2. I didn't manage more than 4 requests/s versus 30 in mysql and hsqldb 2.2. I didn't investigate where the bottleneck was, though.

hsqldb seems to be a nice, feature-rich java database. But there still seems to be a gap between in-memory databases and full file-based databases. I didn't manage to run about 100000 rows in memory with fast startup time and low memory consumption. My next try will be to hand tune these tables.

Installing rpm packages on a deb-system without root access

This blog is a bit of topic, and more or less a note for myself. I just wanted to install OpenOffice3 on my work-machine. Since I have no root-access for that linux-machine, and the administrators are usually busy, I did the following:

• Download the deb-package from openoffice.org (there are internally rpm-files)
  
• Create a private rpm database: mkdir -p /disk1/heiko/rpmdb; rpm --dbpath /disk1/heiko/rpmdb --initdb
• Install the rpm packages without dependancies: rpm --nodeps --dbpath /disk1/heiko/rpmdb --prefix /disk1/heiko/local/OO3.1 --install ooobasis3.1-*.rpm openoffice.org*.rpm

Now I can use openoffice 3.1 by starting /disk1/heiko/local/OO3.1/openoffice.org3/program/soffice

Jetty on a cheap virtual server

In my last blog, I wrote about my first tries on using Jetty to improve the memory footprint for using a cheap (15$) virtual private server with 300MB, running the complete application, including database and webserver. These results have been derived by using benchmarks of the main part of the application, without changing the java settings. Now it is time to come back and talk about the results after running jetty 6.1.14 in production. It should be noted that I had a 100% uptime of the application the first two month of 2009, while still running on tomcat. Shortly after switching to Jetty, the uptime dropped below 98% due to different reasons:

• Running java 1.6.0_11 in client mode was a very bad idea. Jetty simply stalled after 3hours without any error or warning message. I've seen similar behaviour with tomcat 6.0.18, while tomcat 6.0.14 has been running for weeks without problems with the same java version in client mode. I don't know why, but lesson learned: When running a server application, use the server JVM.
• A concurrency bug in my code showed up. The load didn't increase, but jetty seems to be running the code much more concurrent than tomcat did. After fixing the code, that problem was solved.
• A break-in to a different VPS on the same machine caused a lot of problems with extremly slow network and access. After the provider shut down that VPS, everything got better. This was maybe the main cause of the uptime reduction, and it has more to do with VPS handling than with Jetty.

With jetty, I managed to reduce the memory footprint. The application was still running well with new java options. On tomcat, I had to use:
JAVA_OPTS='-server -Xmx96m -Xms96m -Djava.awt.headless=true -XX:+UseConcMarkSweepGC -XX:PermSize=24m -XX:MaxPermSize=24m'
Now I'm using without problems:
JAVA_OPTS='-server -Xmx76m -Xms32m -Djava.awt.headless=true -XX:+UseSerialGC
-XX:MaxPermSize=20m -XX:ReservedCodeCacheSize=16m'
The ReserverdCodeCacheSize is a setting I just found out resently, and it is 46MB by default on a x86 server machine. With some additional tunings for the database reducing its memory requirements from 36MB to 25MB while increasing slightly the cache-hit-ratio from 1% to 2%, I'm now running the application within 220MB, and I'm not afraid of having more than 50hits/s.

Another important adjustment I made to increase the stability was to preallocate a database-connection to each running threads. Usually, a connection pool with 2-5 spare threads would be enough, but on a VPS you never know how long it takes to start a new process. I have seen a simple 'ls' waiting for 10-20s, and the 10s timeout for waiting on a database connection was often timed out, in particular in the morning when people start using my application.

Tomcat 6.0.18 vs Jetty 6.1.14

When I started programming servlets some years ago, Tomcat (5) was the most natural choice to choose as servlet container. It wasn't just the best known container, but it was in addition blessed by SUN as the reference implementation, so it always had the latest compatible features. In addition, Tomcat came bundled with my IDE, so I didn't even think about using a different container. Over the years, the situation has changed a bit. SUN moved to Glassfish/Grisly as reference implementation and I got the impression, that the hype moved from Tomcat to Jetty as the preferred container.

Being lazy, I wouldn't have thought about switching if it wasn't due to some problems I had when I had to upgrade from Tomcat 6.0.14 to 6.0.18. Suddenly, my jsp contained all errors and my cookies threw errors. Both problems were conflicts with the specs: I used 'empty(var)' in my code instead of 'empty (var)' (watch the space) and I didn't uuencode the cookies. But I don't want to see such problems turn up when I just patch because of a security advisory.

Another reason to try something else was the memory consumption. Memory is cheap, but when running servlets on a virtual machine from a hosting provider, price increases close to linearly with memory, since cpu power usually is no issue for people sharing servers.

I decided then to give Jetty a try, since it is known to be small in size and well-suited for embedded applications, which usually have memory limits. My first impression wasn't very good. Download size is quite big, I didn't manage to compile jetty by myself and I couldn't find good documentation. But after some try and error sessions, I managed to get jetty running. The missing documentation turned out to be my biggest problem for setting up all the features I needed for my application, namely: virtual hosts, JNDI database pool, compressed html pages. Searching for documentation on the web didn't help so much since the configuration seems to have changed between Jetty 4, 5 and 6. Chances to get a wrong hint are higher than getting a correct one. The best source of documentation are the javadocs. Here it is clear, that Jetty is more used as embedded container, which needs to be configured on the java level, rather than a standalone application. The configuration file is rather java translated to xml than user-friendly. There were some other inconveniences, but all could be solved.

Finally everything was set up, and I could start testing both containers for performance, and more importantly, for memory consumption. I used the flags which I found out worked best for my application
on tomcat on a two-processor machine:
JAVA_OPTS='-Xmx96m -Xms96m -Djava.awt.headless=true -XX:+UseConcMarkSweepGC -XX:PermSize=24m -XX:MaxPermSize=24m'
and I tested with:
ab -c 20 -n 10000 'http://localhost:8080/dict?searchItem=warum'
I measured the first, the second, the third and the forth 10000 requests, total memory consumption was measured with top-RES.

	#/s 10000	#/s 20000	#/s 30000	#/s 40000	Memory consumption (30000)
jetty -server	224	391	511	512	148m
tomcat -server	248	525	639	33	170m
jetty -client	383	408	414	414	78m
tomcat -client	496	564	543	46	126m

The results show that tomcat memory consumption is more than 20% larger than jetty, while using a client-vm gives another 20%. For performance, the opposite is approximately the case. In addition, after about 35000 requests, tomcat performance drops dramatically, and analysing this further, this is caused mainly by the GC working permanently. I haven't seen that during normal load, where I have approx. 70000 hits a day, so I think this must be some kind of session data, which is kept in tomcat until the session times out after 30min. Talking about real-time usage: I have now jetty running for some days, and I don't recognize any differences. Performance is good, and memory-consumption approximately the same as before. Maybe the most important lesson I have learned: Run with client-vm when running low on memory.

Comparing 64bit and 32bit Linux and Java

I'm running a Tomcat server which needs to allocate large (>1GB) of memory. For that, it would make sense to run Linux and Java in 64bit mode, thus being able to allocate much more than 2GB (Java) or 3.5GB (Linux) of memory. On the other hand, there are a lot of rumours on the net that 64bit Linux uses twice or more the memory. There is a good article including measurements explaining that 64bit java needs approx. 50% more memory for strings and integers.

Here are my results tested on the same machine as virtual xen-hosts, using debian-etch in i386 and amd64 bit mode. The machines got both 256MB memory and no swap:

Action	32bit	64bit
free after boot	219M	217M
file-size c-prog	10M	12M
data-allocation	max 217M	217M
free after data-allocation	224M	224M
java -server	max 208M @ Xmx214M	max 203M @ Xmx234M
java -client	max 209M @ Xmx214M	n.a.

Both the C and the java-program simply allocate some memory. The java-program can be seen at the end of this blog. The java-program tried to allocate 256 * 1MB, until it threw a OutOfMemoryError. The Xmx settings have been adapted so high, that Linux didn't kill the JVM with 'Out of Memory'. It was astonishing to see that the Xmx settings could be set higher on the 64bit jvm, while the maximum available memory within
java was slightly lower.

The results are an assurance that 64bit Linux does not require much more memory than the 32bit Linux. At least not for a server machine with only 64bit libraries installed, and applications which don't require many pointer. Desktop machines may require much more if also the 32bit libraries need to run due to some 32bit only programs. The biggest difference is code size (20%). But gcc is known to have different default options in 32bit and in 64bit mode.

Example of code run to test data allocation:

public class Main {
 public static void main(String[] args) {
  if (args.length == 0) {
   System.err.println("Main: buffers size (in m)");
   System.exit(1);
  }
  int buffers = Integer.parseInt(args[0]);
  int size = Integer.parseInt(args[1]) * 1024 * 1024 / 4;
  java.util.Vector store = new java.util.Vector(size);
  for (int i = 0; i < buffers; i++) {
   store.add(i, new int[size]);
   System.err.println("allocated buffer " + i);
  }
 }
}

Don't be simple - xml

Recently, I'm working with Metamod, a metadata database-application, which can be seen among others on the Damocles website. The metadata of scientifical data is nowadays usually expressed in xml, e.g. DIF or ISO19115. There are several ways to work with xml-files, the best known are maybe DOM, SAX or StAX, often connected to XPath. These come with a lot of commands, implemented in different languages. It is quite of a learning curve to process xml-data, and this is maybe a reason why a lot of people still believe that "just use ASCII" is much better.

Help is coming along by modules like XML::Simple (Perl) or SimpleXML (PHP). These integrate xml nicely into their respective language, converting it to a perl-structure or a php-class. These modules have their right to exist when somebody needs to parse an xml-document written by somebody else and he doesn't want to learn about XML.

Whenever I had to work with XML, it started with something very simple, a good case for the simple modules. But shortly after I have to extend the XML, start with namespaces, modify the original file or something else. And that's where the simple modules fail. The author of XML::Simple has recognized the same problem and written an article how to step up from XML::Simple to DOM/XPATH.

The problem with the "simple modules seems to be old , quoting Einstein: Make everything as simple as possible, but not simpler.

Microbenchmark of SSE in C++ revisited

In my last blog, I described a microbenchmark I've been using, which should explain the performance differences I've been seeing in Fimex when using different sqrt hardware implementations. It found a 20% speedup with SSE, which didn't explain the doubling of speed I've got when using SSE in Fimex.

I managed now to create a working microbenchmark of my vector-normalization program, which shows the differences between using SSE and non-SSE hardware sqrt implementations. While the FP-unit of x86 systems is not IEEE754 aware, the SSE-unit is. I have been using INFINITY for representing undefined values in my code.

Results of the benchmark (in MFLOPS, more is better):

• normal values, no SSE: 237
• normal values, SSE: 280
• INFINITY, no SSE: 12
• INFINITY, SSE: 1380

With a factor of 100 between non-SSE and SSE, I finally understand my performance improvements.