There is a lot of bullshit buzz-speak about “COMET” floating about the internet these days. As best as I can tell, Comet is not an acronym for anything, but rather a play on the term “Ajax” (a related browser communication technique that, like Comet, shares its name with an abrasive cleaner.)

Though it seems that everything written on the subject of Comet is impossibly opaque, the concept is pretty damn simple. Comet is a way around the fact that HTTP is a stateless protocol (i.e., browsers request things, servers serve those things, and then the connection between the two immediately dies.) In order to get new information, browsers have to make requests. Heretofore, this has required applications that need to get frequently updated information to force the browser to “poll” the server at regular intervals to see if new information is available. There was no way for the server to let clients (browsers in this case) know that the new information was available as soon as it became available. Instead, the server had to wait for one of these polling requests to come in again.

This kind of problem is specialized, but important. The utility of being able to “push” data to clients as it becomes available is obvious for live chat, stock tickers, and election results, to name a few examples.

Now, thankfully computers are pretty fast these days, so the polling approach can scale quite nicely; to wit, Campfire, a live web-based chat application produced by our illustrious heros over at 37signals. Campfire does a fine business on a profit-per-request basis, thank you very much. But it does require throwing on some good equipment to support heavy user loads, and more seriously, it still suffers from an annoying lag between when a user types something and when it shows up in the chat window.

When we wrote our live chat application at Lovetastic.com, we chose one precarious solution to the problem to make latency a bit lower. We used Adobe Flash, which exists in most browsers but doesn’t suffer from the same need to communicate only via HTTP. That is, Flash can open stateful sockets with a server and keep the communication flowing both ways. Two problems with this: a) Flash is proprietary and stupid, and b) it requires us to home-brew our own buggy chat server in Python—Ruby’s network programming isn’t so hot. We’re web application programmers and we shouldn’t have to be solving these basic network programming challenges, reinventing the wheel just to get our own low-latency chat. Passing text to clients should be a simple problem, and we don’t see why we should be solving it as “end-user application developers” if you’ll pardon a silly turn of phrase.

Comet has been a long-developing alternative to this Flash approach. Comet, like Ajax, uses features built into Javascript, which is a non-proprietary technology available in nearly all browsers. It uses a little trick built into Javascript to persuade the browser to keep the connection alive as long as possible. In a sense, it’s a bit of a conceptual hack, but I’ve seen it work very well in the case of Lingr, for example. It solves the problem. One might say Ajax itself is a hack too, so whatev.

Trouble is, the writing on Comet available on the internet up until very recently has been scarce at best and absurdly obscurantist at its worst. At the time we set out to create our live chat software, we couldn’t plow through the jargon-laden manifestos to get any practical information out of them. Equally, we weren’t keen on the prospect of having to code either Perl or Java in order to use some of the early comet servers being built.

But fortunately, now it looks like COMET is going to become a whole lot easier to do. This time its thanks to Jan Kneschke, the creator of the webserver lighttpd (which made YouTube possible), with whom we discussed this very prospect at this year’s RailsConf.

Jan has a post over on the lighttpd blog about “mod_mailbox,” which is what he envisions lighttpd’s implementation of COMET to be.

The basic idea is that lighttpd will handle the connections required for COMET, completely decoupling the connection handling from the backend server that handles the logic. So, we’ll be able to write COMET applications that pickup and dropoff messages from the user “mailboxes,” without having to write any of the connection handling that is so hard to do with COMET apps. Also, if he uses the COMET protocol Bayeux, we’ll get the javascript end of the connection handling for free as well.

This will be a huge turning point for COMET… the point at which a developer can code a COMET app without building everything from scratch, a task that is not feasible for most (and was a prospect we certainly didn’t relish ourselves.)

As soon as Jan makes a version available, you can look to Lovetastic.com to be its test-bed.