Latest ESXX Release

Sunday, May 24, 2009

Scaling up with GoGrid

My small GoGrid experiment the other day made me curios. Assume the blog I put online became really popular. How would my deployment cope, and how would I be able to increase capacity?

So I figured I'd script a small benchmark. Assume all visitors land on the Blog's front page, and that half of the visitors click on the first post to read it comments. Finally, assume a third of those add a comment of their own. Translated into a bash script that uses Apache ab, this scenario might look something like this:



function bench {
start=$(date +%s)

${AB} > /dev/null -n 600 -c 60 -e apa ${BASE} &

${AB} > /dev/null -n 300 -c 30 -e apa ${BASE}/posts/1.html &

${AB} > /dev/null -n 100 -c 10 -T application/x-www-form-urlencoded -p ${BASE}/posts/1.html

while [ -n "$(ps | grep -E "^ *(${pid1}|${pid2}|${pid3})")" ]; do
sleep 0.1

end=$(date +%s)

echo "Total time: $(expr ${end} - ${start})"


As you can see, I assume 100 concurrent requests.

On my particular configuration (a 1 core, 0.5 GB RAM virtual GoGrid server in San Francisco and an OpenSolaris laptop in Sweden acting as the client), it turns out that the benchmark takes about 19 seconds per iteration, or 19 ms/request.

It's not too bad, but lets see if we can improve. First of all, we should switch from an embedded to a standalone SQL database. By watching 'top' on our single server, it appears that the ESXX Java process is using up all the CPU; however, since the DB is embedded, we don't know what is the limiting factor; it could be the XSLT transformation, the JavaScript code or the SQL queries.

Since the DB can't easily be scaled horizontally, we deploy a 3 core, 4 GB RAM virtual server and start H2 in server mode on that server, using the following command line:

java -cp h2/bin/h2-1.1.107.jar -Dh2.bindAddress= -tcp -tcpAllowOthers

We then modify the following line in /var/www/ajax-blog/blog.js, to make it connect to the external H2 server instead of using H2 in embedded mode:

var blog = new Blog("jdbc:h2:tcp://;AUTO_RECONNECT=TRUE", "admin", "secret");

Re-running the benchmark reveals that the SQL database sits mostly idle. That's hardly surprising. After all, how much CPU can a few SELECTs from a 100-something row table use? Knowing this, I could have just started the H2 database as a process on the initial web node, but it's always easy to be wise after the event.

Anyway, now we know how to improve the performance. Let's deploy another web front-end, add a load balancer and rerun the benchmark. The result? Almost twice the performance, or 10 ms/request!

And all you need is a web browser, an SSH client and a credit card. The Internet is truly an amazing place.

PS. With four front-ends, we're no longer CPU limited and probably need to tweak the Apache configuration (it's forking like crazy) or ESXX's thread pool settings to obtain further performance improvements.

Friday, May 22, 2009

From localhost to live in 60 minutes using GoGrid

I made an experiment today. The question I wanted to answer was this:
Given a locally developed ESXX application, running on my laptop, how long would it take to go live, assuming you own no servers or Internet connection suitable for such deployment?

For this, I turned to my favourite grid/cloud service, GoGrid. GoGrid is pretty amazing. With just a few clicks in their admin UI, you can create all kinds of servers in their data center, and they will be available online within minutes. Pretty cool stuff, and a perfect match for ESXX, considering the "friendliness" factor!

The app I tested was the Ajax Blog tutorial, which is available as one of the examples in the ESXX distribution.

So I logged in to my GoGrid account and created a "Web/App Server" called "ajax-blog" using the 64-bit CentOS 5.1/Apache 2.2 image. About 15 minutes later, the server was online and I could log in as the root user. After a "yum update" and a reboot, the system was updated to CentOS 5.3 and ready to be configured according to my requirements.

All in all, setting up the "hardware" and the base OS took about two minutes of work and forty minutes of waiting.

First up: software installation. We need to add the ESXX RPM repository, install Java and (unfortunately) we also need to compile mod_fastcgi ourselves. No big deal.

[root@17914_1_20449_109821 ~]# cat > /etc/yum.repos.d/esxx.repo

[root@17914_1_20449_109821 ~]# yum install esxx java-1.6.0-openjdk httpd-devel.x86_64
[root@17914_1_20449_109821 ~]# service esxx start
Starting esxx: [ OK ]
[root@17914_1_20449_109821 ~]# wget
[root@17914_1_20449_109821 ~]# tar xfz mod_fastcgi-2.4.6.tar.gz
[root@17914_1_20449_109821 ~]# cd mod_fastcgi-2.4.6
[root@17914_1_20449_109821 mod_fastcgi-2.4.6]# make -f Makefile.AP2 top_dir=/usr/lib64/httpd/ local-install
[root@17914_1_20449_109821 mod_fastcgi-2.4.6]#

Time to install our ESXX app. We'll create a copy of the Ajax Blog tutorial in /var/www/ajax-blog and tweak the settings a bit:

[root@17914_1_20449_109821 ~]# cp -r /usr/share/doc/esxx/examples/ajax-blog /var/www/
[root@17914_1_20449_109821 ~]# chgrp apache /var/www/ajax-blog/
[root@17914_1_20449_109821 ~]# chmod g+w /var/www/ajax-blog/

Edit /var/www/ajax-blog/blog.js and change the line that creates the 'blog' object to include the full path of the SQL database and a new admin password:

XML.ignoreWhitespace = false;
XML.prettyPrinting = false;

var blog = new Blog("jdbc:h2:/var/www/ajax-blog/Blog", "admin", "secret");

Next, create the file /etc/httpd/conf.d/ajax-blog.conf as follows:
LoadModule fastcgi_module modules/

FastCGIExternalServer /usr/sbin/esxx -host localhost:7654 -pass-header Authorization

# Install handler for all '*.esxx' files
AddType text/x-esxx .esxx
Action text/x-esxx /cgi-bin-esxx
ScriptAlias /cgi-bin-esxx /usr/sbin/esxx

# The Ajax Blog
Alias /blog /var/www/ajax-blog/public
RedirectMatch ^/blog/?$ /blog/index.esxx

Restart httpd, and yeah, that's it. The app is now live and you should be able to visit it at http://YOUR-SERVER-IP/blog/. How's that for friendly deployment?

Thursday, May 21, 2009

ESXX advances into beta

Whoo, long time, no see ... It's been a while since the last blog post, but today, we celebrate the fact that ESXX is no longer considered alpha quality with an all-new look of

I've already blogged about some of the new features in this release, but I just want to mention that there are now two tutorials available in the Wiki. The first one is very basic, while the second one demonstrates how easy it is to create a fully functional, database-backed blog using ESXX, complete with an advanced AJAX administration UI.

It's definately worth a read!