Using Apache's HttpClient on Google App Engine

If you, like me, have tried to use Google's URL Fetch Java API on the Google App Engine, you've probably been disappointed. Sure, it's a small, clean API, but it's totally feature-less. The most advanced thing it supports seems to be ... well, it can follow redirects automatically. Wow. Cookies? Authentication? Forget it.

In ESXX, I use Apache's HttpClient 4, and it works really well. Wouldn't it be nice if you could use HttpClient on the App Engine? Well, now you can. All it takes is a custom connection manager that converts the final requests and feed them into the URL Fetch service, and then feeds the responses back into HttpClient.

You can have a look at the implementation here. It's just two classes, one ClientConnectionManager and one ManagedClientConnection class.

PS. ESXX now runs really well on GAE. Timers, the http, https, mailto, jdbc (in-memory H2) and data URI protocols and HTML parsing, yep, works! Only the dns and ldap are non-functional (they will probably never work). Check out master from Git to try it yourself. Build using ant gae-war.

Updated 2009-12-11: The URIs were double-encoded and query parameters did not work at all. Thanks for pointing this out, Thibaut!

Updated 2010-08-08: The URIs lacked a colon before the optional port number. Thanks for pointing this out, Nello! Also, I changed the license for the two files to LGPLv3.

Updated 2014-11-18: Updated links to Git repo (moved to Github from Berlios).

ESXX on Google App Engine

I got most of ESXX running on Google App Engine today! How cool is that?

Most of the code is in the subversion repository already. I'll try to finish the port as soon as possible and also add seamless support for Google's HTTP client APIs. Once fully checked in, you too can deploy ESXX + your custom JavaScript apps on GAE.

In the meantime, have a look at http://esxx-demo.appspot.com/.

The friendliest server-side JavaScript, now also on Google's servers. Life's sweet.

ThreadPoolExecutor deadlocks

I'm currently trying to get ESXX running on Google's App Engine. One of the problems are that GAE won't let you create background threads or timers, something which most applications, including ESXX, often do.

My initial plan was to switch from a plain ThreadPoolExecutor and Timers to a ScheduledExecutorService. Once done, I would write a GAE-specific, single-threaded version of that class tries to mimic the intended behaviour as good as possible.

The problem is that the ScheduledExecutorService class is a fixed-size thread pool using an unbounded work queue, which just doesn't work in ESXX. Consider this simple example:

import java.util.concurrent.*;

class STPEDeadlock {
  public static void main(String[] args) {
    final ExecutorService exe = 
      new ScheduledThreadPoolExecutor(1, new ThreadPoolExecutor.CallerRunsPolicy());

    System.out.println(Thread.currentThread() + ": Submitting job 1");
    Future j1 = exe.submit(new Runnable() {
        @Override public void run() {
          System.out.println(Thread.currentThread() + ": Submitting job 2");
          Future j2 = exe.submit(new Runnable() {
              @Override public void run() {
                System.out.println(Thread.currentThread() + ": Running job 2");
              }
            });

          try {
            System.out.println(Thread.currentThread() + ": Waiting for job 2");
            j2.get();
          }
          catch (Throwable t) {
            t.printStackTrace();
          }
        }
      });

    try {
      System.out.println(Thread.currentThread() + ": Waiting for job 1");
      j1.get();
    }
    catch (Throwable t) {
      t.printStackTrace();
    }

    System.out.println(Thread.currentThread() + ": All done");
    exe.shutdown();
  }
}

It creates a fixed-size thread pool, and adds and waits for a job. The job, in turn, adds and waits for another job. The problem is that there are no worker threads left, so the second job is just queued and never executed, resulting in a deadlock.

If you replace the executor with

final ExecutorService exe = new ThreadPoolExecutor(1, 1,
                            0L, TimeUnit.MILLISECONDS,
                            new SynchronousQueue(),
                            new ThreadPoolExecutor.CallerRunsPolicy());

then all is fine again. This is the fixed-size executor configuration I use in ESXX. Unfortunately, there is no way to configure a ScheduledExecutorService to match this behavior.

What were the Sun developers thinking??