Shedding Light on Lighttpd

[1]
I know that the ‘A‘ in GLASS stands for Apache, but I have to admit that since last spring, I have been using lighttpd almost exclusively for my performance tests.

It all started last fall, when I noticed that every once in a while, I’d get some pretty noticeable flat spots in the performance graphs when running at rates in excess of 100 requests per second. You can easily see the flat spots in the graph at right (running this setup – 4 gems on a 4 core machine with 10 concurrent siege sessions). Peaking at 200 requests/second, but the graph has big swings.

I originally blamed the the flat spots on “contention,” but I didn’t know where the flat spots were coming from.

In May I started tracking down the source of those flat spots. Over a several days of testing I was able to rule out disk I/O, network I/O, and GemStone internals as the source of the contention. All vital signs on all of the systems involved were flatlined – I used separate machines for siege, Apache, and GLASS.

I finally got around to using tcpdump and I was able to see that the last packet to flow between machines before the flat spot was an HTTP request packet heading into the Apache box. The flat spot ended with a packet heading from the Apache box to the Gemstone box. Pretty clear evidence that Apache was the culprit. Without getting into the internals of Apache, I figured that the contention must be an unfortunate interaction between the MPM worker module (which is multi-threaded) and mod_fastcgi.

I asked our IT guys to install lighttpd and you can see the results in the graph at right (32 gems on an 8 core machine with 180 concurrent siege sessions). In this run we’re peaking at 400 requests/second (twice as many cores), but the performance graph is much tighter (standard deviation of 36 for lighttpd versus 60 with Apache) and best of all, no flat spots. Soooo, if you expect to be hitting rates above 100 requests/second, you should be using lighttpd.

Not only is lighttpd performant, but it is pretty easy to setup as well. It turns out that Miguel Cobá has just published a post on ‘Gemstone/S and FastCGI with lighttpd‘ where he describes how to set up lighttpd for GLASS.

——
[1] Photo: Grandpa’s Shed, Uploaded by a o k on 3 Apr 07, 11.14PM PDT.

Pier 1.0.17 on GLASS

[1]

On Tuesday, Tudor Girba announced the release of Pier 1.0.17 and the creation of a new Pier website. Today (Thursday), I am pleased to announce that Pier 1.0.17 is available on GLASS. I suggest that you keep an eye on the Pier blog for ongoing developments.

If you don’t already have GLASS installed, go on over to the GLASS downloads page for options and instructions. Then follow the instructions here to install the code that includes Pier 1.0.17.

When the final release of GemStone/S 64 version 2.3 is available ([updated 10-06-2008]real soon now) Pier 1.0.17 will be pre-installed. When the appliance based on the final release is made available ([updated 10-21-2008]probably version 1.0 beta 11), it will make a very nice little Pier-in-a-box with built-in persistence and a ready-to-run Pier instance.

Developer Details

When I ported Pier to GLASS I only made one (intentional) change in the feature set (changing PRFile to use MAMemoryFileModel instead of MAExternalFileModel). I didn’t port the package Pier-Squeak-Persistency either for obvious reasons.

By default PRFiles are created using MAExternalFileModel, however, for GLASS, I changed PRFile class>>descriptionFile to use MAMemoryFileModel instead. The rationale is that the GLASS repository is created and initialized on a build machine here at GemStone and any directory structures created during Pier initialization will not be propogated to the final installation. By using MAMemoryFileModel all of the PRFile data is stored in the repository (which happens to be on disk in the final analysis). When you get around to deploying Pier, you will most likely want to use MAExternalFileModel so that you can use Apache or Lighttpd to serve the static files, so you will need to change PRFile class>>descriptionFile to use MAExternalFileModel when you are ready to deploy.

Note – This was my first post using the ‘Blog It!’ feature on Flickr, so some of you may have seen a premature feed in your RSS reader.

——
[1] Photo: Pier Glass Studio, Uploaded by artfull281 on 19 Aug 07, 5.43AM PDT.

GLASS Beta Update: GLASS-dkh.122 & GLASS.230-dkh.162

For the last month or so since the International Smalltalk Developers Conference, I have been busy working on the Single Session per VM project trying to get to the point were I can validate the approach. It was straightforward to make the necessary changes to Seaside, but I literally spent more time trying to figure out how to get session affinity to work with FastCGI in lighttpd.

Once I had the basic prototype working I needed to really hammer the server, but neither siege nor jcrawler does a good enough job (siege doesn’t crawl a site and jcrawler sends concurrent requests for the same Seaside session) so I started work on a Seaside-aware load testing tool called scrawler. Scrawler needs a a real good HTTP client and the HTTP client (HyHTTPClient) in the latest version of Hyper from Bruce Badger is perfect – which explains why I was porting the latest versions of Sport and Hyper to GLASS last week.

Over the weekend Dario Trussardi reported a problem with MAFileDescription which turned into Bug39648 and a few other GemStone-specific issues related to MAExternalFile model. By Monday, I’d resolved Dario’s problem and I figured what the heck, while I was in the neighborhood I might as well merge in the latest Magritte code from Lukas. And while I’m merging code from Lukas, I might as well pick up the latest code for Scriptaculous and Pier.

The ink wasn’t even dry on my Pier merge, when I received mail from Doru, announcing Pier 1.0.17. Talk about synchronicity. Had events (spanning the globe) been conspiring all month long to converge at this moment in time?….Nope, but we do have a pending final release for GemStone/S 64 version 2.3, so I got my giddy up on and updated to Pier 1.0.17.

After the smoke cleared, I ended up including a number of new packages in the GLASS package:

• Pier-Design-tg.5, Pier-Documents-lr.7, Pier-Google-lr.5, Pier-LightBox-dc.6, Pier-Randomizer-lr.4, Pier-TagCloud-lr.11, Pier-Titles-tg.1, Pier-Twitter-lr.7 Pier-Setup-tg.30 to conform with Pier 1.0.17.
• XML-Parser.g-dkh.14 for Pier-Twitter support.

The following bug fixes are also included in this update:

• Bug39648 – FileDirectory>>forceNewFileNamed: doesn’t ensure creation of subdirectories.
• Bug39357 – appliance 1.0 beta10 gems get commit conflicts on first startup.
• Bug39413 – ‘Magritte implements #description: which interferes with subclass creation’.
• fixed duration string generation (RSRSS2) from Philippe Marschall.
• MAFileDescription upload problem reported by Dario Trussardi.

The update is published as GLASS-dkh.122 (2.2.5) and GLASS.230-dkh.162 (2.3 beta 2 or 1.0beta10). These two versions can be used with GemStone-jgf.291 (Squeak client) or later. All of the versions can be found in the GLASS project on GemSource. If you are running with a version of the GLASS package that is earlier than GLASS-dkh.118 or GLASS.230-dkh.146, then you need to follow these instructions first. After loading the GLASS package you should evaluate the following two expressions in a workspace:

1. MADescriptionBuilder initialize.
2. PRDistribution new register.

The first expression picks up some new initialization code and the second expression installs Pier 1.0.17 as a Seaside application.

In GemStone-obi.293 Gerhard Obermann changed the layout of the Transcript Window to allow more buttons, added a Find Substring button, and arranaged for SHIFT-click on the Debug button to empty the Object Log. Personally, I’ve moved to this version of the Squeak client as I think it is an improvement.

If you are running in 2.3 beta 2 or 1.0beta10 then you only need to load GemStone-obi.293 into your Squeak client, save the Squeak image and you are done.

If you are running in a 2.2.5 repository then you have a couple of options:

1. Download the single-click Squeak client for 2.2 (download) from GLASS downloads and then load GemStone-obi.293. RECOMMENDED.
2. Update your existing Squeak image (presumably using GemStone-dkh.283)
   1. start the Squeak client image.
   2. load GemStone-obi.293.
   3. save the image and exit.
   4. copy (or use a symbolic link) the GCI library (location in top pane of the Login Window) to a file named gciForLinux.so, gciForMacintosh.so or gciForWindows.dll, depending upon your client platform.
   5. restart the image and log in to GLASS.

Keep an eye out for the announcement of the final release of GemStone/S 64 version 2.3 [update 10-25-2008] which should be coming soon.

Successful GLASS Tour

Read James’s blog for the full scoop on his tour of User Groups in the Midwest (Columbus Ruby Brigade, Object Technology User’s Group, Detroit Metro Ruby Meetup, Agile Round Table, and Ohio, Indiana, Northern Kentucky PHP Users Group).

Several times the 90 minute tour stretched late into the night. My favorite quote:

At times a few seemed to be paying more attention to their iPhones or laptops than to me, but I discovered later at the social hour that they were Twittering about the possibility of using Seaside on an existing project.

If you are interested in having a GLASS presentation at your user’s group, please send email to James dot Foster at GemStone dot com.

GLASS on Slicehost

James has a post and a couple of screencasts on getting GLASS running on Slicehost.

James Foster on the Road

James Foster has started a blog of his own, Programming Gems (on GemStone) and has kicked things off with an announcement of his September schedule:

• Monday, September 15: Columbus (Columbus Ruby Brigade)
• Tuesday, September 16: Minneapolis/St. Paul (Object Technology User’s Group)
• Wednesday, September 17: Detroit (Detroit Metro Ruby Meetup)
• Thursday, September 18: Cincinnati (Agile Round Table)

Visit his blog for more details.

ESUG 2008

This year 170+ people attended ESUG in Amsterdam. The lecture hall at CWI was packed every day. Visit the media page for links to the presentations from this years conference. Once again, Adriaan van Os, Lukas Renggli, James Robertson and Serge Stinckwich recorded memorable images of the event.

Here are links to the slides for James Foster and my talks. I’ll get a link for Martin McClure’s slides later.

[updated 12/13/2008] Video and audio of Martin’s talk, courtesy of James Robertson.

I have to say that it is quite exciting to see this surge in interest in Smalltalk. Kudos to the ESUG board and conference organizers for investing their time and effort into a successful conference. To ensure the continuing success of the ESUG efforts become a member and show your support for a great organization.

The post conference Seaside Sprint (and here) was also a great success. Thanks to Café Kobalt and the public Library of Amsterdam for (unknowingly) providing the venue for the sprint. It was a pleasure to work on Seaside with such a fine group of programmers.