GLASS Beta Update: GLASS-dkh.108

If you’ve been keeping score, you are probably wondering what happened to the new version of the appliance that I promised at the beginning of the month.

The primary rationale for releasing a new version of the appliance is to make base image code and new primitives available. I did have a hand full of interesting changes queued up awaiting the release of 2.2.5, but the release of 2.2.5 has been held open for bugfixes that might be required by a commercial customer who is taking 2.2.5 into production. While my changes are interesting, they aren’t interesting enough to warrant a release on their own.

Our current plan is to target 2.3 as a Seaside release. 2.3 will include:

1. support for remote breakpoints
2. a handful of base image bugfixes
3. primitive support for UTF8 encoding
4. any other low hanging fruit

I plan on tackling the UTF8 primitives this week, so we can get a 2.3 beta release including a new appliance drop sooner rather than later.

Based on my past prognostications you should know that for me, “Planning is a form of dreaming.”

I’ve just published GLASS-dkh.108 which should be used with GemStone-dkh.258 (both can be found in the GLASS project on GemSource). Here are some of the highlights:

1. Arranged for GemStone-based Transcript messages to be routed to the Squeak-based Transcript.
2. Added GemToGemAnnouncement based on Announcement package from Lukas for handling Gem to Gem signals.
3. Parallel debugging is feature complete:
   1. Debugged continuations can be ‘resumed’.
   2. Remote breakpoints available (need 2.2.5 final and startup script changes).
   3. Profiling works in mulit-vm environment.
   4. Object log improvements (rewrite and ui adjustments).
   5. Object log integrated with Transcript. Transcript used when development vm attached, Object log used otherwise.
4. Several bugfixes (see package history for more details).

In earlier posts, I have talked about remote debugging. I have decided (for now at least) that Parallel debugging is a better name for what we’re doing.

In a nutshell, Parallel debugging is about providing a set of tools for doing debugging that spans multiple vms that are operating in concert. The goal is to make debugging/development in a system that may be composed of hundreds of vms as easy as doing debugging/development in a single vm. Debugging continuations, remote breakpoints and the object log are all components of Parallel debugging support.

I plan on writing a post that will go into more detail about Parallel Development/Debugging with GLASS.

GLASS 101: …Fire

In my post on Simple Persistence for Seaside, I claimed that with the recent enhancements to the Seaside framework in GLASS, it is perfectly okay to use an unprotected class variable to persist shared state. The use of an unprotected class variable will result in occasional commit conflicts, but when a conflict occurs, the framework does an abort and retries the original HTTP request.

So, just how occasional are these commit conflicts and what if I don’t like the idea of using anything uprotecteced. In the last two posts (Ready… and …Aim…), I introduced the tools that you need to answer the question for yourself. In the end it is the behavior of your application that matters the most.

As targets examples, I have written three very simple Seaside examples. Each of the applications is a variation on the theme of shared counters:

• WATally – counter using a class variable, expect retries due to commit conflicts.
• WARcTally – conflict free RcCounter, no commit conflicts.
• WASerial – use an object lock for the production of a unique value, expect retries due to busy/dirty lock.

WATally

With WATally, when a user clicks on the tally link, the current value of the Tally class variable is incremented and the result is stored back intoTally. This is a sure-fire formula for commit conflicts. The following diagram depicts the timeline for three overlapping tally requests:

Request 2 begins before Request 1 is finished, so its commit fails. After a short delay, the request is retried and succeeds. Request 3 also begins before Request 1 is finished, waits a bit and retries, but its retry attempt starts before Request 2 is finished, so the request fails again. Finally on the second retry, the commit for Request 3 succeeds.

If you run jcrawler against WaTally, you can expect to get a handful of Commit failures in the object log (http://localhost/seaside/tools/objectLogin an appliance). I found that using an interval of 1 is best for scaring up Commit failures. Click on the graphic below to see a full width log entry.

If you want to delve into the causes of the commit failure, you can inspect the object log using the following expression:

    System abortTransaction.
    OTRemoteDebugger objectLog asArray inspect.

In this case you’ll find that the commits failed because of a Write-Write conflict on an ‘Association (#Tally->3864)’, which is expected.

It’s worth noting that jcrawler ran at about 12 requests/second for 2 minutes to generate the 38 ‘Commit failures’ and the retry limit was not exceeded. This is validation that ‘retrying requests on commit failure’ is a perfectly safe technique.

WARcTally

With WARcTally, when a user clicks on the tally link, a shared RcCounter instance is incremented. RcCounter is designed to allow concurrent, conflict free increments and decrements. The following diagram depicts the timeline for three overlapping HTTP requests.

Since RcCounters are desinged to be conflict free, each request completes without the need for retries.

If you run jcrawler against WaRcTally, you can expect an empty object log:).

Reduced conflict classes are still the best way to go, as they are designed to prevent commit failures and have undergone a ton of testing.

WASerialNumber

With WASerialNumber, when a user clicks on the tally link, an object lock is obtained on the shared instance of SerialNumber and the value instance variable is incremented and returned. If the lock is either changed or denied a WARetryHTTPRequest is signalled and the HTTP request is retried, much like the retry after a commit conflict for WATally. Object locks are not transactional, so an object lock can be acquired in the middle of a transaction. In this particular example, the lock (once acquired) is kept for the duration of the transaction.

The following diagram depicts the timeline for three overlapping HTTP requests:

Shortly after the processing for Request 1 begins, the SerialNumber object lock is acquired and when Request 1 finishes processing the lock is released.

Request 2 makes its first attempt to acquire the lock, while the lock is held by Request 1, so the lock request is denied. Request 2 signals a WARetryHTTPReqeuest. On the second retry attempt, Request 2 successfully acquires the lock and completes normally.

Request 3 makes its first attempt to acquire the lock after the lock is released by Request 1, so the lock is granted, however, since the transaction for Requst 3 started during the transaction for Request 1 and the SerialNumber instance is dirtied (the value instance variable has a new value), the lock is granted, but with a changed status. In a generic GemStone/S application, an abort would be performed and processing would continue, however, in GLASS/Seaside aborts are not allowed, so a changed lock is treated the same as a denied lock – a WARetryHTTPReqeuest is signalled. On the second attempt to acquire the lock the request is denied, because the lock is held by Request 2. A WARetryHTTPReqeuest is signalled again. Finally, on the second retry, the the lock is granted and the request is completed.

The GLASS/Seaside framework drops an entry in the object log whenever a WARetryHTTPReqeuest is signalled, so you can see the frequency of failed lock attempts. Click in the following image to see the full width log.

If you look closely at the full-eidth image, you will see the two different kinds of ‘Lock not acquired’ messages. ‘SerialNumber lock changed’ and ‘SerialNumber lock denied.’

jcrawler ran at about 10 requests/second for 2 minutes to generate the 72 ‘Lock not acquired’ entries.

Object locking as a technique for avoiding commit conflicts shares the ‘retry on failure’ model of WATally, so it isn’t necessarily superior to ‘retry on commit failure’. It is a superior technique, if you need to protect logical updates where a physical conflict cannot be guaranteed (i.e., updates to different portions of an object graph).

Summary

It certainly looks like the Simple Persistence model of retrying a request on commit failure will stand up under a moderate load. 10 requests/second without retry failures is a pretty good clip. In a real web site, the potential for conflicts is much lower than in these examples since most web page hits are read only, so you can expect to withstand an even higher overall request rate without too much trouble.

It is still a good idea to use reduced conflict classes or object locks in places where you know that concurrent updates can occur, but I think is important to realize that you you don’t have to protect every shared variable from concurrent updates.

Have fun.

GLASS 101: …Aim…

Once you’ve fixed jcrawler so that it can follow dynamic URLs like those used in Seaside, you are ready to aim it at a Seaside application. In this post I’ll give you some pointers for getting started with jcrawler.

Out of the box, jcrawler is written to be run in the jcrawler/dist directory. There are relative paths in the run.sh shell script. That isn’t too convenient, so here’s a bash script that lets you run jcrawler from anywhere:

#!/bin/bash
export JAVA_HOME=<path to java>
export JCRAWLER_HOME=<path to jcrawler>
$JAVA_HOME/bin/java -cp "$JCRAWLER_HOME/:\
$JCRAWLER_HOME/lib/log4j.jar:\
$JCRAWLER_HOME/lib/commons-logging.jar:\
$JCRAWLER_HOME/lib/commons-httpclient.jar:\
$JCRAWLER_HOME/lib/commons-digester.jar:\
$JCRAWLER_HOME/lib/commons-collections.jar:\
$JCRAWLER_HOME/lib/commons-beanutils.jar:\
$JCRAWLER_HOME/dist/jcrawler.jar:\
$JCRAWLER_HOME/lib/htmlparser.jar" com.jcrawler.Main

When you run jcrawler, it expects there to be a crawlerConfig.xml file in a local conf directory (./conf/crawlerConfig.xml). The following is one that I have used for testing against an appliance running at 10.80.21.64 on our internal network. I’ve supplied 4 starting URLs, one for each of the GemStone Examples (randomError, rcTally, tally and serial):

<!--
    Interval (in milliseconds) to invoke a crawl thread.
    There is an HTTP hit every  millisecond.
-->
    <interval>250</interval>

<!--
    Interval (in milliseconds) to invoke a monitor thread.
    Monitor adds new entry in the monitor.log every
    milliseconds
-->
    <monitorInterval>6000</monitorInterval>

<!-- HTTP connection timeout in milliseconds -->
    <connectionTimeout>30000</connectionTimeout>

<!-- Headers to be used by the http client crawler -->
    <headers>
	<header name="User-Agent">Mozilla</header>
	<header name="Cache-Control">no-cache</header>
	<header name="Accept-Language">en-us</header>
    </headers>

<!-- URLs to start crawling from -->
<crawl-urls>
<url>http://10.80.21.64/seaside/examples/GemStone/rcTally/
    </url>
<url>http://10.80.21.64/seaside/examples/GemStone/serial/
    </url>
<url>http://10.80.21.64/seaside/examples/GemStone/tally/
    </url>
<url>http://10.80.21.64/seaside/examples/GemStone/randomError/
    </url>
</crawl-urls>

<!--
    URL patterns (regexps!!!) to allow or deny set of URLs
    permission=true  - these patterns are allowed (anything else is denied)
    permission=false - these patterns are denied (anything else is allowed)
-->
    <url-patterns permission="true">
	<pattern>.*?10.80.21.64.*</pattern>
    </url-patterns>

</settings>

With an interval of 250, jcrawler will run at about 4 requests/second which is fast enough to get started. If you don’t see fireworks at this rate, you can set the interval to zero and let jcrawler fire at will. To really hammer an application you can launch multiple instances of jcrawler.

Here’s a sample object log from one of my runs:

Notice that the report is dominated by entries labeled ‘Lock not acquired – retrying’. A glance at the full width log will show that the retry is due to a ‘Session lock denied: 2075’. A session lock is denied (and the request is retried) if two requests for the same session are received at the same time. This is not surprising given the fact that jcrawler uses a FIFO to store the URLs it scrapes from a page – almost every URL on a single page will have the same session key. When you see errors like this showing up in the object log, you at least know that jcrawler is firing simultaneous requests at Seaside.

As a final note, you should set deployment mode to true (using the Configuration Editor) before pointing jcrawler at your application. If you don’t, you are guaranteed to get some fireworks. While you’re in the Configuration Editor, take a look at the field for setting the Root Component. If the clear link is pressed, the root component for that application is wiped out. Until a new root component is set you will get internal server malfunctions, whenever the WADispatcher tries to launch a new instance of the application. If you don’t set deployment mode to true, jcrawler will eventually find its way into the Configuration Editor and it will eventually hit the clear link.

You are now ready to launch jcrawler at the WATally application and see how our Simple Persistence example fares under load.

GLASS 101: Ready…

Before we point jcrawler at a Seaside application I would like to talk about what you should expect.

To start with, jcrawler is like a bull in a china shop. The algorithm jcrawler uses is not very deterministic nor is it discriminating, but it is thorough, relentless and highly parallel. Given an an initial set of URLs, jcrawler traverses each page and adds the links it finds to its list. Every so often, jcrawler creates a new thread to process another URL from the list. We can depend upon jcrawler to rattle every piece of china in an application and it will rattle more than one piece at a time, so we’d better be ready to deal with wreckage.

Jcrawler will help make your application bullet proof, but at potentially 15 errors/second spread across several vms, there can be a lot of wreckage to sift through.

I added an object log to GLASS a couple of weeks ago and over the last couple of days, I’ve added a Seaside application for viewing and manipulating the object log. Take look at a sample log (my blog is wide-image challenged). It’s not the purtiest page this side of the Mississippi (I am web-design challenged:), but it does the job.

In the object log, the entries labeled ‘– continuation –, represent object log entries that can be debugged via the ‘Debug’ button in the GemStone/S Transcript Window. If you take a peek at the pid column, you will notice that the log entries were generated by two different gems. There are three gems serving HTTP requests in the appliance.

The upshot is that after letting jcrawler hammer on your application, not only do you get an overview of the problems uncovered during the run, but you can open a debugger and investigate the issues that resulted in walkbacks.

I generated the sample object log by manually playing with the randomError application (http://localhost/seaside/examples/GemStone/randomError in the appliance) found in the GemStone Examples project. This little gem generates a simple log entry (‘random error’) or walkback (‘– continuation –‘) 12% of the time. You can also generate different kinds of errors by poking the links in the Error tab of the alltests application (http://localhost/seaside/tests/alltests in the appliance).

If you want to play with the object log, load up the latest version of the GLASS package (GLASS-dkh.103 in the GLASS project – it will also load the GemStone Examples). Poke around in the randErrror application until you get an error then head on over to the object log (http://localhost/seaside/tools/objectLog in the appliance). You can also try the remote debugger from your development image.

Next up we’ll talk a little bit about configuring jcrawler.

GLASS 101: Simple Persistence

In the past, I had talked about how to add persistence to Seaside applications using GLASS, but up until recently, I hadn’t supplied any concrete examples. A month or so ago, the stars aligned, and I took the opportunity to start writing some Seaside examples for GLASS. I figured I would start with a simple example that ignored concurrency issues and then over a series of examples, I’d illustrate the techniques for navigating the treacherous waters of transaction conflicts. I even monkeyed around with jcrawler, so that I’d have a tool that could be used to expose concurrency flaws in the examples.

For the simple persistence example, I decide to copy WACounter, remove the count instance variable and replace it with a class variable reference (Tally) – the simplest example of persistence possible (see WATally in GemStone Examples).

No object lock, no reduced conflict class so look out boys, transaction conflicts are on the menu tonight.

Did I mention that the stars had aligned?

As I was working out the details of an object locking example based on a problem posed by Ken Tries, I realized that when an object lock is denied, the simplest answer is to just retry the HTTP request.

The standard pattern for dealing with a denied object lock in GemStone/S is to wait a bit, abort and attempt to acquire the lock again. If you recall in GLASS, when an HTTP request comes in to a server, we do a beginTransaction (equivalent to an abort), process the request and then do a commit. So, when an object lock is denied while processing a request, we throw an exception, abort the transaction, delay for a bit and retry the request. Each request is handled in its own thread, so while the request is delayed, other requests can be processed by the server. A nice, clean solution that fits very well within the GemStone/Seaside framework.

As I was coding up the solution for object locks (see WASerialNumber and SerialNumber in GemStone Examples) I realized that the same technique (wait a bit and retry the HTTP request) would be the perfect answer for handling transaction conflicts.

So to bring a long story to an end, what started out as a poor example for doing persistence in GLASS has turned into a perfectly good example for doing persistence in GLASS.

All of the rigamarole about needing to avoid transaction conflicts goes right out the window! For high traffic sites or for potentially overlapping requests (i.e., relatively long request processing times), it is still worth using Reduced Conflict classes or object locks to minimize the chance for transaction conflicts. In most cases, though, a simple class variable or global reference will do the trick.

I still want to put WATally under the gun and in my next GLASS 101 post we’ll fire up jcrawler and aim it at WATally. Stay tuned.

ESUG 2008 Call for Contributions

The 16th International Smalltalk Joint Conference is taking place August 25-29 in Amsterdam, the Netherlands. Stéphane Ducasse announced the Call for Contributions, so start making your plans to present or be present at the conference today!

GLASS 101: Disposable Gems, Durable Data

This is the first post in a series of articles that will be covering practical topics for developers working with GLASS. For simplicity’s sake all examples in these articles will be written as if you are using the appliance, however, if you are using a Linux install of GLASS, you should be able to adapt the examples to your specific installation. If you want to delve into more detail you should read the GemStone 101 series of articles or dive into our most excellent documentation.

To start things off, I’d like to drive home the notion of shared, persistent session state for Seaside.

We’ll start with a graphic example where we’ll kill off the web server gems in the middle of a session, restart the gems and observe that the user experience can continue uninterrupted.

To give it a try:

1. Bring up the counter example in your browser (http://localhost/seaside/examples/counter) and click on the ++ link a couple of times.
2. Shut down the Seaside web server gems (using the GLASS Appliance>>GLASS Components>>Gemstone – Gems>>Stop menu item).
3. Click on the ++ link after shutting down the gems and you should get a server error – not surprising.
4. Restart the server gems (using the GLASS Appliance>>GLASS Components>>Gemstone – Gems>>Start menu item).
5. Press the back button in you browser to bring back the cached browser page.
6. Finally, click on ++ link and you will see the the value continues to increment as if nothing had happened.

In GLASS, session state is persistent – cached in the Shared Page Cache (SPC) and stored on disk – along with application data. Being able to share session state and application data via the SPC means that multiple web server vms can service HTTP requests without having to account for session affinity. It also means that web server vms can be brought on and off line as needed without interrupting ongoing sessions.

Probably the most important point is that this is done without requiring any extra work on the part of the developer and it is done very efficiently.

GLASS Beta Update: GLASS-dkh.101

The dogs didn’t let us sleep in this morning, so I’m getting an early start on the day. I’ve been very busy the last two weeks intending to write but getting roped into one thing after another.

I’ve just published GLASS-dkh.101 which should be used with GemStone-dkh.253 (both can be found in the GLASS project on GemSource). Here are some of the highlights:

1. GemStone examples included in the GLASS package. I plan to write descriptions of the examples today!
2. Highlighting added to MCRepositoryBrowser.
3. Started work on MCConfiguration support. I’ve got an OmniBrowser-based tool written and MCConfigurations themselves are mostly working.
4. Improved OmniBrowser tools performance. I did some work to reduce roundtrips and speed up icon calculations.
5. Added MCServerDirectoryRepository. Now you can have directory-based repositories on your client machine (where the Squeak client is running) or on the server machine (where your gems are running).
6. Fixed some bugs, etc. You can look at the history for GLASS-dkh.101 if you want the gory details.

Next week we are planning on releasing a new version of the appliance. Keep an eye on this space for an announcement. If you haven’t played with the appliance yet and have 64bit hardware, drop us an email to get download instructions.

GemStone 101: Avi on GemStone’s Architecture

Avi Bryant has an article article up on his blog where he does a very good job describing GemStone’s architecture. It’s definitely worth a read.