Category Archive

Monty Williams, James Foster, Martin McClure, and myself will be attending the 16th International Smalltalk Joint Conference 2008 put on by ESUG. The conference will be held in in Amsterdam, the Netherlands from August 25-29, 2008. Camp Smalltalk will run August 23-24, 2008.

Martin will be giving a talk entitled “GemStone for Dummies”:

[The] presentation is an introduction to GemStone/S from the viewpoint of the Smalltalk programmer, focusing on the differences from other Smalltalk implementations…

Monty will be giving a short talk on MagLev:

MagLev is a new ruby VM developed by Gemstone…

I’ll be giving another version of “Glass: Share Everything”:

Seaside has been characterized as a “heretical” framework because it breaks many of the widely-accepted “best practices” for web applications, including “share as little state as possible.” With GLASS (GemStone/S, Linux, Apache, Seaside, Smalltalk) GemStone takes this heresy to the next level where “everything is shared” - transparently and persistently….

James will be giving a tutorial “Glass hands on”:

This hands-on tutorial will present Seaside and walk through the process of building an application using GLASS (GemStone, Linux, Apache, Seaside, and Smalltalk). Topics covered include handling user logins, where to put session data vs. application data, building reusable components, styling a web site with CSS, and an introduction to Javascript…

We had an excellent time in Lugano, Switzerland last year and I’m looking forward to another excellent show this year.

Early registration ends on July 15th, so make your plans and register.

For those of you who missed the great conference in Reno this year, you can head over to the STIC site and view the slides for most (if not all) of the presentations.

For the GemStone talks, you can download the slides for my “GLASS: Share Everything” talk or James’ slides from his “Building A Seaside Appplication (with GLASS)” tutorial.

When video and/or audio becomes available, I’ll post another note.

I’ve just published GLASS-dkh.118 (2.2.5) and GLASS.230-dkh.146 (2.3.0 or 1.0beta9). These two versions should be used with GemStone-dkh.279 (Squeak). All three of the versions can be found in the GLASS project on GemSource. If you decide to upgrade to these versions, first load the version on the GemStone-side, commit and logout. Then load GemStone-dkh.279 into your Squeak image and save your Squeak image.

With this update, you will need to do a couple of things to get things to load correctly:

1. Before loading the GLASS package, turn off autoMigrate or evaluate ‘MCPlatformSupport autoMigrate: false.’ The shape and superclass of WAExternalID has been changed in Seaside2.8.2 and the automatic migrate algorithm chokes. As Seaside sessions expire, the old instances will go away.
2. After loading evaluate ‘WAAllTests initialize’ and turn autoMigrate back on.

This release contains the following features:

• Seaside 2.8.2 (Seaside2.8a1-lr.563) merged
• fix bug38366 - blank versions in history list
• fix bug38916: FSSocketServer>>addConnection: and FSSocketServer>>removeConnection: not threadsafe

See GLASS package history for more details.

Between preparations for Smalltalk Solutions and working with Seaside2.9 and 3.0, I’ve been keeping pretty busy. After Smalltalk Solutions, I hope to find time to make another pass at the tools and do a little blogging about GLASS.

It was just over a year ago that we annnounced GLASS at Smalltalk Solutions 2007 (May 2007). We were running on Seaside2.6 and at the time we figured we would be ready for a public beta in about 3 months - ho ho!

By ESUG 2007 (August 2007), we had ported Seaside2.8 to GemStone/S and we had a set of Squeak-based development tools for GLASS. By OOP 2008 (January 2008), we (nearly) had all of the tools implemented in OmniBrowser.

By the end of January we had shipped GemStone/S 64 2.2.4 and made version 1.0beta6 of the appliance available for download to those who had the appropriate hardware and were interested enough to send us an email.

In the last few months, we have addressed commit conflict handling for Seaside, debugging in a multi-vm environment, shipped GemStone/S 64 2.2.5.1 and created 1.0beta9 of the appliance (based on a 2.3 beta).

We’ve had a steady stream of requests for GLASS and now we are prepared to make the beta publicly available. So head on over to the downloads page and get busy! While you are thinking about it join our Beta mailing list, too.

If you are considering putting your GLASS application into production, then you should use version 2.2.5.1. As a released version of GemStone/S 64, 2.2.5.1 has undergone our full QA testing and when future versions of the product become available we will document the procedure for upgrading your GLASS repository to the new version of the product.

I’ve just published GLASS-dkh.117 (2.2.5) and GLASS.230-dkh.140 (2.3.0). These two versions should be used with GemStone-dkh.276GemStone-dkh.278 (Squeak). All three of the versions can be found in the GLASS project on GemSource. If you decide to upgrade to these versions, first load the version on the GemStone-side, commit and logout. Then load GemStone-dkh.276 into your Squeak image and save your Squeak image.

I know, I know, just minutes after announcing the availability of a new version of the appliance I publish a set of new packages….So shoot me! Actually, I don’t anticipate that you’ll be that interested in this particular set of versions, since you might not be able to tell that I’ve made any changes at all: I’ve simply taken the first step in heading towards getting the OB-Enhancements packages and friends loaded into GLASS.

With these versions I have started using a branch of the OB-Standard package instead of the OG-Standard package, so only folks like Liliana and Otto from Finworks will really be interested.

I haven’t had a chance to try loading OB-Enhancements, yet - I’ll save that for next week!

[Updated 5/10/2008 - use version GemStone-dkh.278 (needed for 2.3) and load OG-AllTests-dkh.14 (in the GemStone Tools project) if you want to run the OmniBrowser tests in GemStone)].

I’ll be giving a technical presentation at Smalltalk Solutions entitled “GLASS: Share Everything”:

Seaside has been characterized as a “heretical” framework because it breaks many of the widely-accepted “best practices” for web applications, including “share as little state as possible.” With GLASS (GemStone/S, Linux, Apache, Seaside, Smalltalk) GemStone takes this heresy to the next level where “everything is shared” - transparently and persistently….

According to the preliminary schedule, I will be talking on June 20th from 9:15am to 10:00am.

James Foster will be giving a tutorial entitled “Building a Seaside Application (with GLASS)”:

This hands-on tutorial will present Seaside and walk through the process of building an application using GLASS (GemStone, Linux, Apache, Seaside, and Smalltalk). Topics covered include handling user logins, where to put session data vs. application data, building reusable components, styling a web site with CSS, and an introduction to Javascript.

James’ tutorial is scheduled for June 19th from 1:30pm to 5:00pm.

The conference will be held June 18-21, 2008 in Reno, Nevada, so if you haven’t already signed up, get busy and register!

I’ve just published GLASS-dkh.114 which should be used with GemStone-dkh.270 (both can be found in the GLASS project on GemSource). Here are some of the highlights:

1. AutoCommit and AutoMigrate features added to GLASS tools. If you ever do something that is truly experimental, you can turn off AutoCommit, but in the majority of cases you should be using AutoCommit. With AutoCommit turned on, the system will perform a commit after every UI action that can change state. In the Code and Monticello browsers that means there are commits after virtually every operation. In a workspace, a commit is performed after the expression is evaluated (doIt, printIt, inspectIt, etc.).
2. Seaside2.8.1 merged in (Seaside2.8a1-lr.541).
3. Refactored the Exception hierarchy for GemStone-specific exceptions.

You can get more information by reading the comments in the package history.

GLASS-dkh.114 can be loaded into version 2.2.5 of GemStone/S 64 (including the just annouced version 2.2.5.1). It can also be loaded into version 1.0beta6 of the appliance.

The 2.2.5.1 release of GemStone/S delayed the work towards releasing a new version of the appliance based on 2.3 beta, but we should have a 2.3 beta build done later today and the new version of appliance should be available early next week.

If you’ve been keeping score, you are probably wondering what happened to the new version of the appliance that I promised at the beginning of the month.

The primary rationale for releasing a new version of the appliance is to make base image code and new primitives available. I did have a hand full of interesting changes queued up awaiting the release of 2.2.5, but the release of 2.2.5 has been held open for bugfixes that might be required by a commercial customer who is taking 2.2.5 into production. While my changes are interesting, they aren’t interesting enough to warrant a release on their own.

Our current plan is to target 2.3 as a Seaside release. 2.3 will include:

1. support for remote breakpoints
2. a handful of base image bugfixes
3. primitive support for UTF8 encoding
4. any other low hanging fruit

I plan on tackling the UTF8 primitives this week, so we can get a 2.3 beta release including a new appliance drop sooner rather than later.

Based on my past prognostications you should know that for me, “Planning is a form of dreaming.”

I’ve just published GLASS-dkh.108 which should be used with GemStone-dkh.258 (both can be found in the GLASS project on GemSource). Here are some of the highlights:

1. Arranged for GemStone-based Transcript messages to be routed to the Squeak-based Transcript.
2. Added GemToGemAnnouncement based on Announcement package from Lukas for handling Gem to Gem signals.
3. Parallel debugging is feature complete:
   1. Debugged continuations can be ‘resumed’.
   2. Remote breakpoints available (need 2.2.5 final and startup script changes).
   3. Profiling works in mulit-vm environment.
   4. Object log improvements (rewrite and ui adjustments).
   5. Object log integrated with Transcript. Transcript used when development vm attached, Object log used otherwise.
4. Several bugfixes (see package history for more details).

In earlier posts, I have talked about remote debugging. I have decided (for now at least) that Parallel debugging is a better name for what we’re doing.

In a nutshell, Parallel debugging is about providing a set of tools for doing debugging that spans multiple vms that are operating in concert. The goal is to make debugging/development in a system that may be composed of hundreds of vms as easy as doing debugging/development in a single vm. Debugging continuations, remote breakpoints and the object log are all components of Parallel debugging support.

I plan on writing a post that will go into more detail about Parallel Development/Debugging with GLASS.

In my post on Simple Persistence for Seaside, I claimed that with the recent enhancements to the Seaside framework in GLASS, it is perfectly okay to use an unprotected class variable to persist shared state. The use of an unprotected class variable will result in occasional commit conflicts, but when a conflict occurs, the framework does an abort and retries the original HTTP request.

So, just how occasional are these commit conflicts and what if I don’t like the idea of using anything uprotecteced. In the last two posts (Ready… and …Aim…), I introduced the tools that you need to answer the question for yourself. In the end it is the behavior of your application that matters the most.

As targets examples, I have written three very simple Seaside examples. Each of the applications is a variation on the theme of shared counters:

• WATally - counter using a class variable, expect retries due to commit conflicts.
• WARcTally - conflict free RcCounter, no commit conflicts.
• WASerial - use an object lock for the production of a unique value, expect retries due to busy/dirty lock.

WATally

With WATally, when a user clicks on the tally link, the current value of the Tally class variable is incremented and the result is stored back intoTally. This is a sure-fire formula for commit conflicts. The following diagram depicts the timeline for three overlapping tally requests:

Request 2 begins before Request 1 is finished, so its commit fails. After a short delay, the request is retried and succeeds. Request 3 also begins before Request 1 is finished, waits a bit and retries, but its retry attempt starts before Request 2 is finished, so the request fails again. Finally on the second retry, the commit for Request 3 succeeds.

If you run jcrawler against WaTally, you can expect to get a handful of Commit failures in the object log (http://localhost/seaside/tools/objectLogin an appliance). I found that using an interval of 1 is best for scaring up Commit failures. Click on the graphic below to see a full width log entry.

If you want to delve into the causes of the commit failure, you can inspect the object log using the following expression:

    System abortTransaction.
    OTRemoteDebugger objectLog asArray inspect.

In this case you’ll find that the commits failed because of a Write-Write conflict on an ‘Association (#Tally->3864)’, which is expected.

It’s worth noting that jcrawler ran at about 12 requests/second for 2 minutes to generate the 38 ‘Commit failures’ and the retry limit was not exceeded. This is validation that ‘retrying requests on commit failure’ is a perfectly safe technique.

WARcTally

With WARcTally, when a user clicks on the tally link, a shared RcCounter instance is incremented. RcCounter is designed to allow concurrent, conflict free increments and decrements. The following diagram depicts the timeline for three overlapping HTTP requests.

Since RcCounters are desinged to be conflict free, each request completes without the need for retries.

If you run jcrawler against WaRcTally, you can expect an empty object log:).

Reduced conflict classes are still the best way to go, as they are designed to prevent commit failures and have undergone a ton of testing.

WASerialNumber

With WASerialNumber, when a user clicks on the tally link, an object lock is obtained on the shared instance of SerialNumber and the value instance variable is incremented and returned. If the lock is either changed or denied a WARetryHTTPRequest is signalled and the HTTP request is retried, much like the retry after a commit conflict for WATally. Object locks are not transactional, so an object lock can be acquired in the middle of a transaction. In this particular example, the lock (once acquired) is kept for the duration of the transaction.

The following diagram depicts the timeline for three overlapping HTTP requests:

Shortly after the processing for Request 1 begins, the SerialNumber object lock is acquired and when Request 1 finishes processing the lock is released.

Request 2 makes its first attempt to acquire the lock, while the lock is held by Request 1, so the lock request is denied. Request 2 signals a WARetryHTTPReqeuest. On the second retry attempt, Request 2 successfully acquires the lock and completes normally.

Request 3 makes its first attempt to acquire the lock after the lock is released by Request 1, so the lock is granted, however, since the transaction for Requst 3 started during the transaction for Request 1 and the SerialNumber instance is dirtied (the value instance variable has a new value), the lock is granted, but with a changed status. In a generic GemStone/S application, an abort would be performed and processing would continue, however, in GLASS/Seaside aborts are not allowed, so a changed lock is treated the same as a denied lock - a WARetryHTTPReqeuest is signalled. On the second attempt to acquire the lock the request is denied, because the lock is held by Request 2. A WARetryHTTPReqeuest is signalled again. Finally, on the second retry, the the lock is granted and the request is completed.

The GLASS/Seaside framework drops an entry in the object log whenever a WARetryHTTPReqeuest is signalled, so you can see the frequency of failed lock attempts. Click in the following image to see the full width log.

If you look closely at the full-eidth image, you will see the two different kinds of ‘Lock not acquired’ messages. ‘SerialNumber lock changed’ and ‘SerialNumber lock denied.’

jcrawler ran at about 10 requests/second for 2 minutes to generate the 72 ‘Lock not acquired’ entries.

Object locking as a technique for avoiding commit conflicts shares the ‘retry on failure’ model of WATally, so it isn’t necessarily superior to ‘retry on commit failure’. It is a superior technique, if you need to protect logical updates where a physical conflict cannot be guaranteed (i.e., updates to different portions of an object graph).

Summary

It certainly looks like the Simple Persistence model of retrying a request on commit failure will stand up under a moderate load. 10 requests/second without retry failures is a pretty good clip. In a real web site, the potential for conflicts is much lower than in these examples since most web page hits are read only, so you can expect to withstand an even higher overall request rate without too much trouble.

It is still a good idea to use reduced conflict classes or object locks in places where you know that concurrent updates can occur, but I think is important to realize that you you don’t have to protect every shared variable from concurrent updates.

Have fun.

Once you’ve fixed jcrawler so that it can follow dynamic URLs like those used in Seaside, you are ready to aim it at a Seaside application. In this post I’ll give you some pointers for getting started with jcrawler.

Out of the box, jcrawler is written to be run in the jcrawler/dist directory. There are relative paths in the run.sh shell script. That isn’t too convenient, so here’s a bash script that lets you run jcrawler from anywhere:

#!/bin/bash
export JAVA_HOME=<path to java>
export JCRAWLER_HOME=<path to jcrawler>
$JAVA_HOME/bin/java -cp "$JCRAWLER_HOME/:\
$JCRAWLER_HOME/lib/log4j.jar:\
$JCRAWLER_HOME/lib/commons-logging.jar:\
$JCRAWLER_HOME/lib/commons-httpclient.jar:\
$JCRAWLER_HOME/lib/commons-digester.jar:\
$JCRAWLER_HOME/lib/commons-collections.jar:\
$JCRAWLER_HOME/lib/commons-beanutils.jar:\
$JCRAWLER_HOME/dist/jcrawler.jar:\
$JCRAWLER_HOME/lib/htmlparser.jar" com.jcrawler.Main

When you run jcrawler, it expects there to be a crawlerConfig.xml file in a local conf directory (./conf/crawlerConfig.xml). The following is one that I have used for testing against an appliance running at 10.80.21.64 on our internal network. I’ve supplied 4 starting URLs, one for each of the GemStone Examples (randomError, rcTally, tally and serial):

<!--
    Interval (in milliseconds) to invoke a crawl thread.
    There is an HTTP hit every  millisecond.
-->
    <interval>250</interval>

<!--
    Interval (in milliseconds) to invoke a monitor thread.
    Monitor adds new entry in the monitor.log every
    milliseconds
-->
    <monitorInterval>6000</monitorInterval>

<!-- HTTP connection timeout in milliseconds -->
    <connectionTimeout>30000</connectionTimeout>

<!-- Headers to be used by the http client crawler -->
    <headers>
	<header name="User-Agent">Mozilla</header>
	<header name="Cache-Control">no-cache</header>
	<header name="Accept-Language">en-us</header>
    </headers>

<!-- URLs to start crawling from -->
<crawl-urls>
<url>http://10.80.21.64/seaside/examples/GemStone/rcTally/
    </url>
<url>http://10.80.21.64/seaside/examples/GemStone/serial/
    </url>
<url>http://10.80.21.64/seaside/examples/GemStone/tally/
    </url>
<url>http://10.80.21.64/seaside/examples/GemStone/randomError/
    </url>
</crawl-urls>

<!--
    URL patterns (regexps!!!) to allow or deny set of URLs
    permission=true  - these patterns are allowed (anything else is denied)
    permission=false - these patterns are denied (anything else is allowed)
-->
    <url-patterns permission="true">
	<pattern>.*?10.80.21.64.*</pattern>
    </url-patterns>

</settings>

With an interval of 250, jcrawler will run at about 4 requests/second which is fast enough to get started. If you don’t see fireworks at this rate, you can set the interval to zero and let jcrawler fire at will. To really hammer an application you can launch multiple instances of jcrawler.

Here’s a sample object log from one of my runs:

Notice that the report is dominated by entries labeled ‘Lock not acquired - retrying’. A glance at the full width log will show that the retry is due to a ‘Session lock denied: 2075′. A session lock is denied (and the request is retried) if two requests for the same session are received at the same time. This is not surprising given the fact that jcrawler uses a FIFO to store the URLs it scrapes from a page - almost every URL on a single page will have the same session key. When you see errors like this showing up in the object log, you at least know that jcrawler is firing simultaneous requests at Seaside.

As a final note, you should set deployment mode to true (using the Configuration Editor) before pointing jcrawler at your application. If you don’t, you are guaranteed to get some fireworks. While you’re in the Configuration Editor, take a look at the field for setting the Root Component. If the clear link is pressed, the root component for that application is wiped out. Until a new root component is set you will get internal server malfunctions, whenever the WADispatcher tries to launch a new instance of the application. If you don’t set deployment mode to true, jcrawler will eventually find its way into the Configuration Editor and it will eventually hit the clear link.

You are now ready to launch jcrawler at the WATally application and see how our Simple Persistence example fares under load.

Before we point jcrawler at a Seaside application I would like to talk about what you should expect.

To start with, jcrawler is like a bull in a china shop. The algorithm jcrawler uses is not very deterministic nor is it discriminating, but it is thorough, relentless and highly parallel. Given an an initial set of URLs, jcrawler traverses each page and adds the links it finds to its list. Every so often, jcrawler creates a new thread to process another URL from the list. We can depend upon jcrawler to rattle every piece of china in an application and it will rattle more than one piece at a time, so we’d better be ready to deal with wreckage.

Jcrawler will help make your application bullet proof, but at potentially 15 errors/second spread across several vms, there can be a lot of wreckage to sift through.

I added an object log to GLASS a couple of weeks ago and over the last couple of days, I’ve added a Seaside application for viewing and manipulating the object log. Take look at a sample log (my blog is wide-image challenged). It’s not the purtiest page this side of the Mississippi (I am web-design challenged:), but it does the job.

In the object log, the entries labeled ‘– continuation –, represent object log entries that can be debugged via the ‘Debug’ button in the GemStone/S Transcript Window. If you take a peek at the pid column, you will notice that the log entries were generated by two different gems. There are three gems serving HTTP requests in the appliance.

The upshot is that after letting jcrawler hammer on your application, not only do you get an overview of the problems uncovered during the run, but you can open a debugger and investigate the issues that resulted in walkbacks.

I generated the sample object log by manually playing with the randomError application (http://localhost/seaside/examples/GemStone/randomError in the appliance) found in the GemStone Examples project. This little gem generates a simple log entry (’random error’) or walkback (’– continuation –’) 12% of the time. You can also generate different kinds of errors by poking the links in the Error tab of the alltests application (http://localhost/seaside/tests/alltests in the appliance).

If you want to play with the object log, load up the latest version of the GLASS package (GLASS-dkh.103 in the GLASS project - it will also load the GemStone Examples). Poke around in the randErrror application until you get an error then head on over to the object log (http://localhost/seaside/tools/objectLog in the appliance). You can also try the remote debugger from your development image.

Next up we’ll talk a little bit about configuring jcrawler.

In the past, I had talked about how to add persistence to Seaside applications using GLASS, but up until recently, I hadn’t supplied any concrete examples. A month or so ago, the stars aligned, and I took the opportunity to start writing some Seaside examples for GLASS. I figured I would start with a simple example that ignored concurrency issues and then over a series of examples, I’d illustrate the techniques for navigating the treacherous waters of transaction conflicts. I even monkeyed around with jcrawler, so that I’d have a tool that could be used to expose concurrency flaws in the examples.

For the simple persistence example, I decide to copy WACounter, remove the count instance variable and replace it with a class variable reference (Tally) - the simplest example of persistence possible (see WATally in GemStone Examples).

No object lock, no reduced conflict class so look out boys, transaction conflicts are on the menu tonight.

Did I mention that the stars had aligned?

As I was working out the details of an object locking example based on a problem posed by Ken Tries, I realized that when an object lock is denied, the simplest answer is to just retry the HTTP request.

The standard pattern for dealing with a denied object lock in GemStone/S is to wait a bit, abort and attempt to acquire the lock again. If you recall in GLASS, when an HTTP request comes in to a server, we do a beginTransaction (equivalent to an abort), process the request and then do a commit. So, when an object lock is denied while processing a request, we throw an exception, abort the transaction, delay for a bit and retry the request. Each request is handled in its own thread, so while the request is delayed, other requests can be processed by the server. A nice, clean solution that fits very well within the GemStone/Seaside framework.

As I was coding up the solution for object locks (see WASerialNumber and SerialNumber in GemStone Examples) I realized that the same technique (wait a bit and retry the HTTP request) would be the perfect answer for handling transaction conflicts.

So to bring a long story to an end, what started out as a poor example for doing persistence in GLASS has turned into a perfectly good example for doing persistence in GLASS.

All of the rigamarole about needing to avoid transaction conflicts goes right out the window! For high traffic sites or for potentially overlapping requests (i.e., relatively long request processing times), it is still worth using Reduced Conflict classes or object locks to minimize the chance for transaction conflicts. In most cases, though, a simple class variable or global reference will do the trick.

I still want to put WATally under the gun and in my next GLASS 101 post we’ll fire up jcrawler and aim it at WATally. Stay tuned.

This is the first post in a series of articles that will be covering practical topics for developers working with GLASS. For simplicity’s sake all examples in these articles will be written as if you are using the appliance, however, if you are using a Linux install of GLASS, you should be able to adapt the examples to your specific installation. If you want to delve into more detail you should read the GemStone 101 series of articles or dive into our most excellent documentation.

To start things off, I’d like to drive home the notion of shared, persistent session state for Seaside.

We’ll start with a graphic example where we’ll kill off the web server gems in the middle of a session, restart the gems and observe that the user experience can continue uninterrupted.

To give it a try:

1. Bring up the counter example in your browser (http://localhost/seaside/examples/counter) and click on the ++ link a couple of times.
2. Shut down the Seaside web server gems (using the GLASS Appliance>>GLASS Components>>Gemstone - Gems>>Stop menu item).
3. Click on the ++ link after shutting down the gems and you should get a server error - not surprising.
4. Restart the server gems (using the GLASS Appliance>>GLASS Components>>Gemstone - Gems>>Start menu item).
5. Press the back button in you browser to bring back the cached browser page.
6. Finally, click on ++ link and you will see the the value continues to increment as if nothing had happened.

In GLASS, session state is persistent - cached in the Shared Page Cache (SPC) and stored on disk - along with application data. Being able to share session state and application data via the SPC means that multiple web server vms can service HTTP requests without having to account for session affinity. It also means that web server vms can be brought on and off line as needed without interrupting ongoing sessions.

Probably the most important point is that this is done without requiring any extra work on the part of the developer and it is done very efficiently.

The dogs didn’t let us sleep in this morning, so I’m getting an early start on the day. I’ve been very busy the last two weeks intending to write but getting roped into one thing after another.

I’ve just published GLASS-dkh.101 which should be used with GemStone-dkh.253 (both can be found in the GLASS project on GemSource). Here are some of the highlights:

1. GemStone examples included in the GLASS package. I plan to write descriptions of the examples today!
2. Highlighting added to MCRepositoryBrowser.
3. Started work on MCConfiguration support. I’ve got an OmniBrowser-based tool written and MCConfigurations themselves are mostly working.
4. Improved OmniBrowser tools performance. I did some work to reduce roundtrips and speed up icon calculations.
5. Added MCServerDirectoryRepository. Now you can have directory-based repositories on your client machine (where the Squeak client is running) or on the server machine (where your gems are running).
6. Fixed some bugs, etc. You can look at the history for GLASS-dkh.101 if you want the gory details.

Next week we are planning on releasing a new version of the appliance. Keep an eye on this space for an announcement. If you haven’t played with the appliance yet and have 64bit hardware, drop us an email to get download instructions.

It’s an absolutely beautiful day today. Every February we get a week of sunny, fifty degree weather and it looks like this is the week. It is so nice to see the sun! Over the holiday weekend we were able to take our dogs for a run in a nearby dog park. It’s a real treat to see a Saluki in full flight.

Besides playing with dogs, I’ve spent some time playing with and cleaning up some things in GLASS. I’ve published a new version of the GLASS package (GLASS-dkh.88) and here are some of the hightlights:

1. I’m getting happier with the remote debugging factility - the steps for debugging are greatly simplified and I’ve even replaced the Hyper button on the GemStone Transcript with a Debug button that will open a debugger on the first continuation in the RcQueue (you’ll need to load GemStone-dkh.240 into your Squeak image for this feature). When you’re in production (#deploymentMode == true), the debug continuations are unconditionally added to the queue.
2. In the same vein as remote debugging, I’ve added a consolidated object log where you can dump a LogEntry into an RcQueue from any one of the gems serving Seaside requests and view the logged objects in the comfort of your own development image. No need to muck around in N different logs to look for stack traces or important error messages.
3. You may have noticed that in the past I have been somewhat vague whenever the subject of commit failures was brought up. Now, when you hit a commit failure, we drop a transactionConflicts Dictionary into the object log (see the comment in System class transactionConflicts for a description of its contents), and simply retry the HTTP request. If retry fails 10 times in a row, we throw in internal error.

I’ll be writing a couple of posts in the next week or so to provide more details.

James Robertson has posted the audio for James Foster’s talk on GLASS at the Ottawa Carleton Smalltalk Users Group last week.

David Buck recorded the audio.

If you went to Winterlude instead of attending James’ talk, you’ve got another chance to hear him talk about Seaside.

Many thanks to David Buck, James Robertson and Cincom.

jcrawler is a good tool for load testing Seaside applications - in theory. Unfortunately, it takes a little bit of work to modify jcrawler to get it to the point where it can be used to load test Seaside applications. So here’s my story…

The Story

In the last week I’ve started putting together some examples of persistence for GLASS (I’ll write a post about them when I’m happy with the examples). As I have mentioned before, you need to use different techniques to manage concurrency in GemStone/S, because you will be using multiple vms to serve web pages and Semaphores can’t be used. The examples illustrate several of the techniques that you can use to avoid transaction conflicts. As part of the exercise, I needed to find a way to test for transaction conflicts.

In order to create a transaction conflict, you need to have two web requests hit your web server at exactly the same time. I’ve used siege in the past for load testing, but siege uses constant URLs, not too useful for banging arbitrary URLs buried in the depths of your Seaside application.

To effectively beat on a Seaside application (especially if you want to expose concurrency bugs) you need a load tester that will crawl through your site, pick up the dynamically generated URLs and feed them back into the mix.

I knew that WAPT had been used by several folks for Seaside Load Tests, but I didn’t see site crawling mentioned in the feature list for WAPT. Beside that I’m doing my work on Linux boxes, so a Windows-only tool would not be convenient.

Without trying too hard, I found a site that listed a ton of Web Test Tools and up near the top of the of the Load and Performance Test Tools section there was a listing for jcrawler:

An open-source stress-testing tool for web apps; includes crawling/exploratory features. User can give JCrawler a set of starting URLs and it will begin crawling from that point onwards, going through any URLs it can find on its way and generating load on the web application. Load parameters (hits/sec) are configurable via central XML file; fires up as many threads as needed to keep load constant; includes self-testing unit tests. Handles http redirects and cookies; platform independent.

Just the ticket, huh? Well, if it was that easy, I wouldn’t be writing a blog post would I? Haha!

The Work

I grabbed the download from SourceForge and proceeded to build jcrawler.

You need ant, too. But that’s easily fixed.

The build completed and I was ready to slam my Seaside apps and its only been a couple of minutes! But run.sh failed:

Exception in thread “main” java.lang.UnsupportedClassVersionError: com/jcrawler/Main (Unsupported major.minor version 49.0)

It turns out that you must use JDK 5.0. Another download and some monkey business with my environment variables:

export JAVA_HOME=/home/dhenrich/jdk1.5.0_14
export PATH=$JAVA_HOME/bin:$PATH

and we’re off to the races. I launched jcrawler against:

http://172.16.172.138/seaside/examples/persistence/rcTally

a variant on WACounter running on my copy of the appliance.

The Problem

Things appeared to running okay. jcrawler was spinning away dumping log entries like the following to stdout (sorry about the line wrapping):

2568 [THREAD#93 CREATED 10:40:01::602] INFO com.jcrawler.UrlFetcher - Fetching URL http://172.16.172.138/seaside/examples/persistence/serial?
_s=lKtxmydVdpOJAjpt&_k=ggcZGlQC&1

However, as I interactively poked at rcTally, I noticed that jcrawler wasn’t hitting the + + or - - links, because the shared value was not getting updated.

After an excruciating amount of debugging, I noticed that the URLs extracted from the web page contained the sequence ‘&‘ instead of ‘&‘…geez it has been just about as hard to get WordPress to display the dang ‘&‘ string in my post (can’t use rich editor) as it was to find the problem in jcrawler (sorry about line wrapping):

http://172.16.172.138/seaside/examples/persistence/serial?
_s=lKtxmydVdpOJAjpt&_k=ggcZGlQC&1

The Fix

1. Download an HTML Parser from SourceForge.
2. Copy the jars from the HTML Parser into jcrawler:
   

   cd /home/dhenrich/htmlparser1_5/lib
   cp *.jar /home/dhenrich/jcrawler/lib

3. Edit the jcrawler source and insert the following line after line 120 in com/jcrawler/UrlFetcher.java (in the jcrawler src directory) to convert the encoded HTML:
   

   content = org.htmlparser.util.Translate.decode(content);

4. add htmlparser.jar to the list of jars in run.sh (in the jcrawler misc directory).
5. Rebuild jcrawler and you are off to the races!

The Payoff

At the end of the day, you’ve got yourself a version of jcrawler, that can be used to randomly poke around in the nooks and crannies of your Seaside application and give it a pretty thorough workout.

As I work on the GemStone examples, I’ll learn more about jcrawler’s quirks and features, but for now it does pretty much what I need.

If there’s another load tester out there that can crawl through a Seaside website, I’d appreciate hearing about it.

Version 1.0beta6 of the appliance is rolling off the assembly line and that new software smell is once again wafting through the hallways of our Beaverton office. If you are using an appliance that is older than 1.0beta5, then you should contact us to get download instructions for 1.0beta6.

If you haven’t played with the appliance yet and have 64bit hardware, drop us an email to get download instructions, you should also read the Terse Guide to GLASS Tools.

On the development front, I am busy getting a port of SourceSource 2 running on GemStone. SourceSource 2 is Philippe Marschall’s port of SqueakSource to Seaside2.8 and Magritte. When I’m done I think that this will be a good example of what types of things need to be done to port a Squeak Seaside app to GemStone.

Oh, and don’t forget to sign up for our GLASS Beta mailing list.

Like the Siberian Express, James Foster will be blowing into the Northeast early next week. James will be talking about GLASS:

While the Seaside framework elegantly addresses HTML generation and application flow-of-control issues, it still leaves challenges for the developer–including persistence, multi-user coordination, and scaling. With typical solutions (including object-relational mapping, external files, and multiple images) the “pure objects” experience of Smalltalk is compromised. In this presentation we will demonstrate GLASS (GemStone, Linux, Apache, Seaside, and Smalltalk), a stack (analogous to LAMP) that provides a robust environment for deploying sophisticated, dynamic web applications that can scale.

Having lived for a while in Fargo, North Dakota, James is no stranger to winter weather and I understand that he is prepared to hire a team of huskies if necessary. So, if you are starting to get cabin fever and a trip to the Seaside sounds good, then it’s time to get out your Mukluks and mush on over to one of the following Smalltalk User Group meetings:

• Toronto Smalltalk User Group - February 4, 2008
• Ottawa Carleton Smalltalk Users Group - February 5, 2008
• New York City Smalltalk Users Group - February 6, 2008

BTW, James just might have a copy or two of the latest appliance.

David Buck has an announcement for the next Ottawa Smalltalk Users Group meeting where James Foster will be talking about GLASS. From David’s post:

While the Seaside framework elegantly addresses HTML generation and application flow-of-control issues, it still leaves challenges for the developer–including persistence, multi-user coordination, and scaling. With typical solutions (including object-relational mapping, external files, and multiple images) the “pure objects” experience of Smalltalk is compromised. In this presentation we will demonstrate GLASS (GemStone, Linux, Apache, Seaside, and Smalltalk), a stack (analogous to LAMP) that provides a robust environment for deploying sophisticated, dynamic web applications that can scale.

If you’re in the area, you should plan on heading over to the meeting.

Updated: [in a comment, Charles reminded me that James is hitting a trio of Smalltalk User Groups in Feburary, thanks Charles]:

• Toronto Smalltalk User Group - February 4, 2008
• Ottawa Carleton Smalltalk Users Group - February 5, 2008
• New York City Smalltalk Users Group - February 6, 2008

Ken Treis has started work on a MockGemStone package that provides implementations for a couple of the Rc classes discussed in GemStone 101:Transaction Conflicts. The package can be loaded into a Squeak image and the classes can be used to develop a Seaside application in Squeak that will run in both GemStone and Squeak environments.

To the extent that any tool set is ever actually done, I’m glad to say that the GemStone tools for GLASS are done. It has been a long haul, but I’m feeling real good about the state of the tools. I will write up a Terse Guide to the GLASS Tools in the next day or so to give you an idea of what to expect.

In order for you to actually get a look at the new tools you’ll have to wait for Version 2.2.4 of GemStone/S 64 or download a new version of the Seaside Appliance.

Version 2.2.4 is due to be released this Friday. We’re in the final stages of QA for this release and so far we are on track to make the date.

Once 2.2.4 goes out the door, we will build the new version of the appliance. It should be available next week. If you have been using an older version of the appliance, it is highly recommended that you download the new version. If you haven’t played with the appliance yet and have 64bit hardware drop us an email to get download instructions.

Between the holidays and time off to wrestle with a stubborn cold, December was a short, but busy month.

Version 2.2.4 of GemStone/S 64 is due to be released around the middle of January, so I’ve been scrambling to make changes to the base classes that are needed for OmniBrowser and Seaside.

We will also be cranking out a new rev of the GLASS appliance around the same time, so I’ve been working to get all of the image-based tests to pass and to clean up some of the Undeclared Symbols that have been floating around. Infrastructure improvements for the most part. With this rev of the GLASS appliance, we’ll still be in limited beta (i.e., if you have 64 bit hardware and want to take GLASS for a spin, drop us an email), but we are getting very close to making the beta available for download.

The big news is that we’ve finally got OmniBrowser-based Monticello tools, what a nice Xmas gift that was! We brought Avi down to Portland for a week and he nailed the Monticello tools. He did his development in Squeak, so we have made the results available to Damien Cassou (via the Package Universe) and he should be including the new tools in the beta Squeak development image. The OB-Monticello package is available in Colin’s OmniBrowser repository. I will be making a handful of changes in the next week or so as I get the new Monticello tools working in GemStone/S, but the plan is to have them ready for the next rev of the appliance. When that work is done, I’ll refresh the information in the Package Universe.

Finally, don’t forget that OOP 2008 in Munich (January 21-25, 2008) is coming up.

This is the second article in the GemStone 101 series. If you haven’t already done so, I recommend that you read GemStone 101: Transactions and Unlimited GemStone VMs in every Garage? ….and a Stone in every Pot before reading this post. It also wouldn’t hurt to take a gander at The Pillars of Concurrency for background on concurrency issues.

Transaction Conflicts

Today we’re going to delve a bit into the world of transaction conflicts. In GemStone/S a transaction conflict occurs when two or more sessions attempt to commit changes to the same object. A commit conflict is an error condition and in most cases you will end up aborting the transaction (for a discussion transaction conflicts in more detail read Chapter 6.1 in the GemStone/S Programming Guide). The best medicine for transaction conflicts is to avoid them.

[Updated: 3/17/2008] See GLASS 101: Simple Persistence for an alternative to conflict avoidance.

Object Locks

Avoiding transaction conflicts is really not that much different than dealing with concurrent access to a shared data structure in a single vm when multiple processes are running. In Squeak, you would simply protect shared data from concurrent access by using a Semaphore and a critical: block. In GemStone/S you can’t use Semaphores to protect persistent shared data, since each session is running in separate operating system processes (potentially on a different machine). You can, however, use object locks to control concurrent access to persistent shared data (See Chapter 6.3 in the GemStone/S Programming Guide for complete details on manipulating object locks).

Object locks are session based. Once a session obtains an object lock, other sessions will not be allowed to obtain a lock on the same object until the lock is released. For protecting access to shared data, you can arrange to release the lock on transaction boundaries (commit or abort), so that the access to the shared data is protected for the duration of the transaction.

If you are denied an object lock, you must wait for the lock to be released via polling or by using a variant of the object lock called an application lock. A request for an application lock will automatically block until the lock is available. Application locks are more convenient than standard object locks, but there are restrictions on the number of application locks that you can use.

Using a object lock on an object to protect access to shared data is a perfectly valid technique, but it should be used sparingly to avoid getting into potential deadlock situations and to avoid delays waiting for the lock to be released. Object locks are also relatively expensive to use, as they involve interaction with the stoned process.

Reduced Conflict Classes

As an alternative to using object locks to avoid transaction conflicts, GemStone/S provides a set of reduced conflict classes (See Chapter 6.4 in the GemStone/S Programming Guide for complete details on reduced conflict classes), that in many cases allow one to perform concurrent, conflict-free updates on the same object:

• RcCounter
• RcQueue
• RcIdentityBag
• RcKeyValueDictionary

In GemStone/S, a transaction conflict occurs when the same object is modified by two different sessions, even when the modifications are not logically inconsistent. For example, adding two different objects to an IdentityBag is not logically inconsistent, but will result in a transaction conflict if performed by two different sessions.

The reduced conflict classes were added to GemStone/S to allow logically consistent updates to be made to the same object by different sessions, while avoiding transaction conflicts.

Logically inconsistent operations on reduced conflict classes will still result in a failed transaction. For example, two sessions adding the same key to a RcKeyValueDictionary is logically inconsistent and will result in a failed transaction.

Reduced conflict classes should be used as your first line of defense in avoiding transaction conflicts.

RcCounter

A separate object is allocated for tracking the contributions from each session. The value of the RcCounter is calculated when you request its value by traversing over the contributions for each session.

RcQueue

For RcQueue, entries are timestamped and added to a separate collection for each producer session. A consumer session then ‘removes’ the elements from the queue keeping track of the elements removed without actually removing them from the producer session’s collection. You can avoid conflicts under the following conditions:

• Any number of sessions read objects in the queue at the same time.
• Any number of sessions add objects to the queue at the same time.
• One session removes an object from the queue while any number of sessions are adding objects.

RcIdentityBag

For RcIdentityBag, the contents are managed by keeping an ‘adds Bag’ and ‘removals Bag’ for each session. When you enumerate the contents, a cache of the bag (in transient session state) is created by adding and removing objects recorded for each session. You can avoid conflicts under the following conditions:

• Any number of sessions read objects in the bag at the same time.
• Any number of sessions add objects to the bag at the same time.
• One session removes an object from the bag while any number of sessions are adding objects.
• Any number of sessions remove objects from the bag at the same time, as long as no more than one of them tries to remove the last occurrence of an object.

RcKeyValueDictionary

For RcKeyValueDictionary, reduced conflict operation is achieved by recording adds and removes in a RedoLog. If a transaction conflict is encountered, objects within the dictionary are selectively aborted and the add and remove operations are replayed and a second commit is attempted. You can avoid conflicts under the following conditions:

• Any number of sessions read values in the dictionary at the same time.
• Any number of sessions add keys and values to the dictionary at the same time, unless a session tries to add a key that already exists.
• Any number of sessions remove keys from the dictionary at the same time, unless more than one session tries to remove the same key at the same time.
• Any number of users perform any combination of these operations.

Reduced Conflicts for Indexed Collections

In a future GemStone 101 article, I will go into a little more detail about indexed collections, suffice to say that if you need ordered access to, or need to perform queries on elements in a shared collection, you can use an RcIdentityBag with an RC equality index or two to reduce the likelihood of transaction conflicts.

Concurrency Control in Seaside

In GLASS we have modified the Seaside framework to use both object locks and reduced conflict classes to avoid transaction conflicts.

As I have written before, when a request comes into a GLASS vm, the framework does a beginTransaction before passing the HTTP request to the Seaside application code and the framework does a commitTransaction before passing the HTTP response to the user’s web browser.

Before we do the beginTransaction, we acquire a write lock on the WASession instance to ensure that no two vms are handling a request for the same WASession concurrently. The write lock is then released when the commitTransaction is performed. The object lock is the correct construct to use here, because the intent is to block other sessions from running concurrently. As an intended side effect, this also means that all session-specific data is protected from transaction conflicts.

Each WAApplication maintains a couple of handler dictionaries that are protected by a Semaphore in the standard Seaside code base. For GLASS, we use instances of the reduced conflict dictionary class RcKeyValueDictionary. In this case it isn’t necessary to completely block access to the dictionaries while they are updated. It is sufficient to ensure that the dictionaries are logically consistent at the end of the transaction - reduced conflict classes are the right answer in this case.

For the data structures (that aren’t already protected by the session lock) in your Seaside application, you can apply these tests to help decide what technique to use to avoid transaction conflicts:

• use write locks when concurrent updates to a data structure cannot be tolerated
• use reduced conflict classes when your usage pattern conforms to the restrictions imposed by the particular reduced conflict class

Back in October, I said that I was going heads down on the tools and I really wasn’t kidding. During November I didn’t blog very much, but I did make some major progress.

First, I finished off the port of Magritte that was started by Lukas and Isaiah Perumalla.

I dipped my toe into the OmniBrowser-based Monticello tools, but before long it was obvious that I needed to migrate to the latest version of OmniBrowser. Firstly, we’d be able to make some of the very nice features, that have been recently added to OmniBrowser, available to GemStone developers. Secondly, we’d be able to use the current development image available from Damien Cassou as the basis for the GemStone/S tools. eliminating the need to build a specific Squeak image for GemStone.

While merging the latest OmniBrowser changes into the GemStone code-base, I got fed up with the inspector for stack contexts in the debugger. I’m pretty happy with the resulting context inspector, but then I’m probably a little biased. The new context inspector gives you an overview of the arguments, temps and receiver state that includes names and values, something like what you can get with Dolphin (but not quite that nice). The inspector is slaved to the debugger, so the contents are updated on each step or whenever you select a new context.

During Thanksgiving week I took some vacation and spent quality down time with my wife, family and doggies.

Based on feedback from a beta user, we finally decided to bite the bullet. With the next release of the GemStone (version 2.2.4), the GemStone/S compiler will support Squeak-style curly brace Array constructors, as well as method pragmas. At ESUG, Damien Cassou questioned why we didn’t just add the curly brace Array constructor to our compiler - he was right, it just took us a while to get around to it.

Well November began with OmniBrowser-based Monticello tools at the top of my list and it ended with them still at the top, but a fair amount of necessary work was done along the way.

James Foster is spanning the globe with a full complement of GLASS presentations in the next couple of months.

In December, James is putting on a GemStone and Seaside workshop at the Smalltalk Conference of Argentina in Buenos Aires, Argentina - December 10-12, 2007.

In January, James and Monty Williams will be representing GemStone at OOP 2008 in Munich, Germany - January 21-25, 2008.

Then in February James will hit a trio of STUGs on the East Coast:

• Toronto Smalltalk User Group - February 4, 2008
• Ottawa Carleton Smalltalk Users Group - February 5, 2008
• New York City Smalltalk Users Group - February 6, 2008

Mark your calendar and make plans to visit with James when he comes calling in your neighborhood.

James Foster of GemStone will be going to the First Smalltalk Conference of Argentina (December 10, 11 and 12, 2007). The conference is being held in Buenos Aires at the University of Buenos Aires. As of November 14th, there are over 170 registrations.

James will lead two half-day workshops on Wednesday, December 12. The morning session will be an “Introduction to GLASS: Seaside with Transparent Persistence.” The afternoon session will be “Scaling GLASS: Growing Your System.”

If you are ready to Tango with GLASS, then you should head on down to sunny Buenos Aires!

We’re having an Indian Summer in Oregon this year. The trees are alive with vibrant reds and bright yellows against a backdrop of Douglas Fir green and a bright blue sky. Balmy, clear weather in Oregon, in October - Fantastic! As the sun climbs up over the Cascades and lights up the maples in the parking lot, I’m thinking back on the fact that I spent the entire summer working on the tools for GLASS and it looks like I’ll be spending my Indian Summer doing the same thing.

While I was playing with GemStone/S and Seaside, Isaiah Perumalla spent some time porting Magritte to GemStone/S. Isaiah (with some help from Lukas Renggli) got to the point where all but a few of the tests were passing. Some GemStone-specific work was needed and I’ve stepped in to finish off the port, but the bulk of the work was done by Isaiah and Lukas.

In the midst of working through the remaining Magritte tests, I decided that I would get breakpoints working in the OmniBrowser tools. I’ve got them working now, but I now have to add a Breakpoint Browser to the list of tools that need to be implemented.

Also, Otto Behrens and Liliana Ivan from Finworks found time to extend Shout to do syntax highlighting for GemStone/S Smalltalk, which is very nice.

If you have 64 bit hardware and the time to explore GLASS right now, visit our website and drop us an email to let us know that you’re ready to rumba.

Meanwhile, I’m going heads down on the tools.

Philippe Marschall has anounced the final version of Seaside 2.8. You can find the final GemStone/S version on GemSource (Seaside2.8g1-dkh.522.mcz). Lukas Renggli has additional information .

Many of you have heard that there are GemStone/S applications running in production with thousands of vms and that other applications are doing thousands of commits per second. I bet you have been wondering what kind of performance you could get running Seaside on GemStone/S - I certainly was:)

Seriously, I spent most of the summer working on tools and with the exception of a few small tests, I didn’t take the time to focus on performance. When I came back from ESUG, I decided to run some scaling tests.

[For those of you who want to see the results and move on with your surfing, here are a couple of links: Goals, Results, Conclusions].

Background

From the tests that I had run over the summer it appeared that Apache was a limiting factor when trying to run at rates above 30 requests per second. I’d also seen some anomalies in the i/o on Linux, were we’d get flat spots in our performance graphs that appeared to be related to file system buffer flushing. If you have read the post on transactions, then you know that we write tranlog records on every commit (page request), so disk i/o can be a limiting factor.

With the help of our IS guy, it turned out that to get the best performance from Apache, we needed to use the MPM worker module. With the MPM worker module turned on, Apache performance fell way off the radar.

The issue with the i/o anomalies that we observed in Linux has not been as easy to resolve. I spent some time tuning GemStone/S to make sure that GemStone/S wasn’t the source of the anomaly. Finally our IS guy was able to reproduce the anomaly and he ran into a few other folks on the net that have observed similar anomalies.

At this writing we haven’t found a solution to the anomaly, but we are pretty optimistic that it is resolvable. We’ve seen different versions of Linux running on similar hardware that doesn’t show the anomaly, so it is either a function of the kernel version or the settings of some of the kernel parameters. As soon as we figure it out we’ll let you know.

For the purposes of these performance tests, I was able to work around the i/o anomaly by putting extents on raw partitions. In nearly all of the tests, the Shared Page Cache (SPC) is sized large enough to hold the entire working set for the test. Consequently there was very little read activity and the system was able to write dirty pages from the SPC fast enough so that random i/o to the raw extent partitions didn’t affect test results.

Goals

I had three goals in mind when I ran this set of tests.

1. demonstrate anticipated performance of the GLASS appliance.
2. demonstrate production performance for the Web Edition.
3. demonstrate performance potential beyond the Web Edition.

For the GLASS applicance tests I wanted to illustrate what you could expect if you installed the VMWare image on a machine without paying particular attention to disk configurations. To simulate the GLASS applicance, I simply ran the tests using file-based tranlogs.

For the production-scale performance of the Web Edition, I wanted to illustrate what you could expect if you paid attention to the disk configuration (i.e., created some raw partitions and had a box with a minimum of 4 disk spindles).

I also wanted to illustrate what kind of performance improvements you could see if you were to add additional SPC or increase the number CPUs available.

Finally, based on a comment where Ramon Leon suggested I include some ’speed comparisons between GemStone and Squeak’, I’ve included runs against a Squeak vm.

Test Strategy

I decided to base the performance tests on the Seaside Counter example. Since the Counter has dead-simple render logic and no significant application state, it is the perfect application for measuring baseline Seaside performance. For GemStone that means that when running tests against the Counter, we’ll be getting performance numbers that include the overhead of persisting and sharing session state (about 250 objects or 50k bytes per request).

Over the course of the summer I ran across siege, “an http regression testing and benchmarking utility” that is very easy to use. It can smack the heck out of a Web Application without putting too much of a load on the system running the test. It also provides some very basic stats about how your app withstood the barrage. Response Time, Transaction rate, and Concurrency being the most interesting.

Siege basically arranges to fire a number of concurrent requests at a given URL. In benchmark mode, as soon as a response is recieved another request is launched in its place. In this mode you can force an application to its knees - very nice for finding bottlenecks. In internet mode, siege waits a random amount of time before firing off the follow-on request, making for a simulation of what the end users of your application may experience.

I ran my tests using the URL ‘http://penny:8000/seaside/examples/counter’, which, as many of you Seasiders out there will recognize, means that a new Seaside session is created on each hit. For benchmarking purposes, that’s just fine - a little extra load never hurts. In the future, I plan to run some scaling tests that measures intra-session performance.

For quite a while now, I have felt that it was important for folks to plan on running multiple vms when they go into production using GLASS. For the best overall response times, one should plan on running at least one vm per concurrent request, which in practice should be 10 or more. For the benchmark tests, the Counter application is cpu bound, so when siege goes about slamming the web server into the ground the cpus are pegged and cpu contention between the processes becomes a factor. It turns out that with 10 vms running on a single cpu, there is a whole lot of contention going on. So after playing around a bit, I settled on using 5 vms for the benchmark tests while running with 10 concurrent requests. This combo gave good performance numbers in the single core tests while in the 4 core test when all 4 cpus were redlined we would minimize the amount of contention. I also ran a couple of internet tests using a single CPU and 20 vms to confirm that in the wild you could afford to run with more than 5 vms without suffering a performance hit. Finally I ran a benchmark test with 100 concurrent requests to see how the system behaved when it was being slashdotted.

Test Setup

For the hardware I used 3 machines foos, toronto, and penny.

Penny is a 2.6Ghz AMD Opteron, with 2 dual core cpus, running SUSE Enterprise 10, with 8Gb of ram. Penny was used to host Apache and Siege. We had a dedicated 1Gbs ethernet connection between penny and toronto.

Foos is a 2.4Ghz Intel, with 1 dual core cpu, running SUSE Enterprise 10, with 2Gb of ram and 2 disk drives (no raw partitions). Foos was used to simulate the GLASS appliance performance.

Toronto is a 2.2Ghz AMD Opteron with 2 dual core cpus, running SUSE Enterprise 10, with 8Gb of ram and 5 disk drives (raw partitions installed on 3 of the partitions). Toronto was used to simulate a typical production machine.

Siege was pointed at the Apache instance listening on port 8000. In addition to the mpm_worker_module, mod_proxy_balancer was used to round-robin requests to the various vms. GemStone was running with version Seaside2.8g1-dkh.490 of Seaside.

For the Squeak tests I used the latest development image from Damien Cassou (sq3.9-7067dev07.10.1), the 3.9 vm and loaded Seaside2.8a1-lr.492. I pointed siege directly at the Squeak image (both running on Toronto).

I did not enforce exclusive use on any of the machines (penny and toronto are shared by other folks in the company) during the tests. But between running most of the tests multiple tim