Category Archive

Monty Williams, James Foster, Martin McClure, and myself will be attending the 16th International Smalltalk Joint Conference 2008 put on by ESUG. The conference will be held in in Amsterdam, the Netherlands from August 25-29, 2008. Camp Smalltalk will run August 23-24, 2008.

Martin will be giving a talk entitled “GemStone for Dummies”:

[The] presentation is an introduction to GemStone/S from the viewpoint of the Smalltalk programmer, focusing on the differences from other Smalltalk implementations…

Monty will be giving a short talk on MagLev:

MagLev is a new ruby VM developed by Gemstone…

I’ll be giving another version of “Glass: Share Everything”:

Seaside has been characterized as a “heretical” framework because it breaks many of the widely-accepted “best practices” for web applications, including “share as little state as possible.” With GLASS (GemStone/S, Linux, Apache, Seaside, Smalltalk) GemStone takes this heresy to the next level where “everything is shared” - transparently and persistently….

James will be giving a tutorial “Glass hands on”:

This hands-on tutorial will present Seaside and walk through the process of building an application using GLASS (GemStone, Linux, Apache, Seaside, and Smalltalk). Topics covered include handling user logins, where to put session data vs. application data, building reusable components, styling a web site with CSS, and an introduction to Javascript…

We had an excellent time in Lugano, Switzerland last year and I’m looking forward to another excellent show this year.

Early registration ends on July 15th, so make your plans and register.

For those of you who missed the great conference in Reno this year, you can head over to the STIC site and view the slides for most (if not all) of the presentations.

For the GemStone talks, you can download the slides for my “GLASS: Share Everything” talk or James’ slides from his “Building A Seaside Appplication (with GLASS)” tutorial.

When video and/or audio becomes available, I’ll post another note.

It was just over a year ago that we annnounced GLASS at Smalltalk Solutions 2007 (May 2007). We were running on Seaside2.6 and at the time we figured we would be ready for a public beta in about 3 months - ho ho!

By ESUG 2007 (August 2007), we had ported Seaside2.8 to GemStone/S and we had a set of Squeak-based development tools for GLASS. By OOP 2008 (January 2008), we (nearly) had all of the tools implemented in OmniBrowser.

By the end of January we had shipped GemStone/S 64 2.2.4 and made version 1.0beta6 of the appliance available for download to those who had the appropriate hardware and were interested enough to send us an email.

In the last few months, we have addressed commit conflict handling for Seaside, debugging in a multi-vm environment, shipped GemStone/S 64 2.2.5.1 and created 1.0beta9 of the appliance (based on a 2.3 beta).

We’ve had a steady stream of requests for GLASS and now we are prepared to make the beta publicly available. So head on over to the downloads page and get busy! While you are thinking about it join our Beta mailing list, too.

If you are considering putting your GLASS application into production, then you should use version 2.2.5.1. As a released version of GemStone/S 64, 2.2.5.1 has undergone our full QA testing and when future versions of the product become available we will document the procedure for upgrading your GLASS repository to the new version of the product.

I assume that most of you have heard about MagLev which is a product that brings GemStone/S techology to the Ruby world. It is no secret that MagLev leverages a large chunk of the GemStone/S code base, but what hasn’t been emphasized is the fact that we are planning to release a new version (version 3.0) of the GemStone/S 64 product that is based upon the same MagLev vm technology.

Development for the new vm actually began last summer with an effort to speed up the Smalltalk vm by generating native code. Along the way features have been added, like the ability to support the Ruby language:). For Smalltalk execution, it looks like the 3.0 vm will be 1.5 to 2 times faster than the 2.0 vm.

We are shooting at the end of the year for the release of ‘GemStone/S 64 3.0′ and we will have more product details at Smalltalk Solutions.

For Seaside fans, you will be interested to know that I am in the middle of porting Seaside2.9 to version 3.0. I’m down to 4 failed and 15 errors in the Seaside unit tests.

GemStone has received a few requests for training classes and we are close to having enough students to offer classes (we must have at least 3 students per class).

If you are interested in GemStone training in the next 3 months or so, please contact Norm Green (send mail and Norm will get back to you). Course descriptions are shown below.

Read the rest of this entry »

After a few hitches in our giddyup, we are pleased to announce the availability of the 1.0Beta9 GLASS appliance. The appliance is based upon GemStone/S 64 Version 2.3 Beta 1. We have published the download instructions (join our Beta mailing list) or you can send mail to request download instructions. To make sure that your hardware can run VMWare, double check the hardware requirements.

In the future, when I announce the availability of a new GLASS package in a GLASS Beta Update post we plan on making a new version of the appliance available as well.

The fabled 2.3 beta release of GemStone/S 64 is finally available for download (contact Monty Williams or send mail for download instructions) .

The beta includes the following features:

1. Primitive support for UTF8 encoding (100x faster than Smalltalk-based algorithm).
2. Primitive support for HTML encoding.
3. Thread local support for GsProcess and continuations. Thread locals are being used in Seaside2.9.
4. Improved ProfMonitor output code moved from GLASS-only to base image.

The beta is shipped with GLASS.230-dkh.130 and GemStone-dkh.270 (both can be found in the GLASS project on GemSource). GLASS.230-dkh.130 is equivalent to GLASS-dkh.114, except for references to the UTF8 and HTML encoding primitives.

A version of the appliance based upon 2.3 beta 1 will be available shortly.

I’ll be giving a technical presentation at Smalltalk Solutions entitled “GLASS: Share Everything”:

Seaside has been characterized as a “heretical” framework because it breaks many of the widely-accepted “best practices” for web applications, including “share as little state as possible.” With GLASS (GemStone/S, Linux, Apache, Seaside, Smalltalk) GemStone takes this heresy to the next level where “everything is shared” - transparently and persistently….

According to the preliminary schedule, I will be talking on June 20th from 9:15am to 10:00am.

James Foster will be giving a tutorial entitled “Building a Seaside Application (with GLASS)”:

This hands-on tutorial will present Seaside and walk through the process of building an application using GLASS (GemStone, Linux, Apache, Seaside, and Smalltalk). Topics covered include handling user logins, where to put session data vs. application data, building reusable components, styling a web site with CSS, and an introduction to Javascript.

James’ tutorial is scheduled for June 19th from 1:30pm to 5:00pm.

The conference will be held June 18-21, 2008 in Reno, Nevada, so if you haven’t already signed up, get busy and register!

We just announced the release of GemStone/S 64 version 2.2.5.1.

2.2.5.1 includes a Seaside extent based on GLASS-dkh.114. For our Seaside users we fixed a bug related to remote breakpoints, so that with 2.2.5.1, you can set/debug/resume remote breakpoints.

As I mentioned for 2.2.5, most Seaside users should wait for the 2.3 beta appliance. 2.2.5.1 was created primarily for those Seaside users that need a production release for development as well as deployment.

Yesterday we announced the release of GemStone/S 64 version 2.2.5. Check out the Release Notes for more information.

From a GLASS perspective we fixed only a hand full of bugs. The 2.2.5 release primarily contains features and bugfixes for a commercial customer.

For GLASS users, I recommend that you wait for the 2.3 beta that is scheduled to be available at the end of this week.

Avi Bryant has an article up on his blog where he does a very good job describing GemStone’s architecture. It’s definitely worth a read.

Yesterday we announced the release of GemStone/S 64 version 2.2.4. Check out the Release Notes for more information.

From a GLASS perspective, there were a handful of changes made to the base product:

• Curly brace Array constructors added to the Smalltalk compiler.
• Support for Squeak-style Pragmas.
• Notification mechanism for abortTransaction, beginTransaction, commitTransaction, and transactionMode:. (see the class SessionMethodTransactionBoundaryPolicy).
• Notification mechanism for sessionStart (see the class SystemLoginNotification).

You need the changes in version 2.2.4 in order to use the GLASS tools. As of today, you can load the Monticello package GLASS-dkh.57.mcz into GemStone/S from the GLASS repository (you will need GemStone-dkh.234.mcz loaded into your Squeak image as well).

In addition to the base support for Monticello and Seaside, GLASS-dkh.57.mcz includes several packages from the SeasideExamples, Scriptaculous, SeasideAsync, Magritte, and Pier. All of the unit tests are passing for the packages. You can look at the version history for GLASS-dkh.57.mcz to get a blow by blow account of the changes.

In the last week before shipping 2.2.4, we found a bug that affects doing a proceed from a halt or breakpoint when debugging Seaside applications. In order to fix the bug, we had to make changes to the C executables. Since 2.2.4 is being released to customers for production applications, we didn’t want to delay 2.2.4, so the bugfix wasn’t included.

After the 2.2.4 release, we did a beta build for version 2.2.5 and included the bugfix. The new version of the appliance (which is in the final phases of production) is being built with the 2.2.5 beta. I recommend that everyone use the 2.2.5 beta for Seaside development.

David Buck has an announcement for the next Ottawa Smalltalk Users Group meeting where James Foster will be talking about GLASS. From David’s post:

While the Seaside framework elegantly addresses HTML generation and application flow-of-control issues, it still leaves challenges for the developer–including persistence, multi-user coordination, and scaling. With typical solutions (including object-relational mapping, external files, and multiple images) the “pure objects” experience of Smalltalk is compromised. In this presentation we will demonstrate GLASS (GemStone, Linux, Apache, Seaside, and Smalltalk), a stack (analogous to LAMP) that provides a robust environment for deploying sophisticated, dynamic web applications that can scale.

If you’re in the area, you should plan on heading over to the meeting.

Updated: [in a comment, Charles reminded me that James is hitting a trio of Smalltalk User Groups in Feburary, thanks Charles]:

• Toronto Smalltalk User Group - February 4, 2008
• Ottawa Carleton Smalltalk Users Group - February 5, 2008
• New York City Smalltalk Users Group - February 6, 2008

Ken Treis has started work on a MockGemStone package that provides implementations for a couple of the Rc classes discussed in GemStone 101:Transaction Conflicts. The package can be loaded into a Squeak image and the classes can be used to develop a Seaside application in Squeak that will run in both GemStone and Squeak environments.

This is the second article in the GemStone 101 series. If you haven’t already done so, I recommend that you read GemStone 101: Transactions and Unlimited GemStone VMs in every Garage? ….and a Stone in every Pot before reading this post. It also wouldn’t hurt to take a gander at The Pillars of Concurrency for background on concurrency issues.

Transaction Conflicts

Today we’re going to delve a bit into the world of transaction conflicts. In GemStone/S a transaction conflict occurs when two or more sessions attempt to commit changes to the same object. A commit conflict is an error condition and in most cases you will end up aborting the transaction (for a discussion transaction conflicts in more detail read Chapter 6.1 in the GemStone/S Programming Guide). The best medicine for transaction conflicts is to avoid them.

[Updated: 3/17/2008] See GLASS 101: Simple Persistence for an alternative to conflict avoidance.

Object Locks

Avoiding transaction conflicts is really not that much different than dealing with concurrent access to a shared data structure in a single vm when multiple processes are running. In Squeak, you would simply protect shared data from concurrent access by using a Semaphore and a critical: block. In GemStone/S you can’t use Semaphores to protect persistent shared data, since each session is running in separate operating system processes (potentially on a different machine). You can, however, use object locks to control concurrent access to persistent shared data (See Chapter 6.3 in the GemStone/S Programming Guide for complete details on manipulating object locks).

Object locks are session based. Once a session obtains an object lock, other sessions will not be allowed to obtain a lock on the same object until the lock is released. For protecting access to shared data, you can arrange to release the lock on transaction boundaries (commit or abort), so that the access to the shared data is protected for the duration of the transaction.

If you are denied an object lock, you must wait for the lock to be released via polling or by using a variant of the object lock called an application lock. A request for an application lock will automatically block until the lock is available. Application locks are more convenient than standard object locks, but there are restrictions on the number of application locks that you can use.

Using a object lock on an object to protect access to shared data is a perfectly valid technique, but it should be used sparingly to avoid getting into potential deadlock situations and to avoid delays waiting for the lock to be released. Object locks are also relatively expensive to use, as they involve interaction with the stoned process.

Reduced Conflict Classes

As an alternative to using object locks to avoid transaction conflicts, GemStone/S provides a set of reduced conflict classes (See Chapter 6.4 in the GemStone/S Programming Guide for complete details on reduced conflict classes), that in many cases allow one to perform concurrent, conflict-free updates on the same object:

• RcCounter
• RcQueue
• RcIdentityBag
• RcKeyValueDictionary

In GemStone/S, a transaction conflict occurs when the same object is modified by two different sessions, even when the modifications are not logically inconsistent. For example, adding two different objects to an IdentityBag is not logically inconsistent, but will result in a transaction conflict if performed by two different sessions.

The reduced conflict classes were added to GemStone/S to allow logically consistent updates to be made to the same object by different sessions, while avoiding transaction conflicts.

Logically inconsistent operations on reduced conflict classes will still result in a failed transaction. For example, two sessions adding the same key to a RcKeyValueDictionary is logically inconsistent and will result in a failed transaction.

Reduced conflict classes should be used as your first line of defense in avoiding transaction conflicts.

RcCounter

A separate object is allocated for tracking the contributions from each session. The value of the RcCounter is calculated when you request its value by traversing over the contributions for each session.

RcQueue

For RcQueue, entries are timestamped and added to a separate collection for each producer session. A consumer session then ‘removes’ the elements from the queue keeping track of the elements removed without actually removing them from the producer session’s collection. You can avoid conflicts under the following conditions:

• Any number of sessions read objects in the queue at the same time.
• Any number of sessions add objects to the queue at the same time.
• One session removes an object from the queue while any number of sessions are adding objects.

RcIdentityBag

For RcIdentityBag, the contents are managed by keeping an ‘adds Bag’ and ‘removals Bag’ for each session. When you enumerate the contents, a cache of the bag (in transient session state) is created by adding and removing objects recorded for each session. You can avoid conflicts under the following conditions:

• Any number of sessions read objects in the bag at the same time.
• Any number of sessions add objects to the bag at the same time.
• One session removes an object from the bag while any number of sessions are adding objects.
• Any number of sessions remove objects from the bag at the same time, as long as no more than one of them tries to remove the last occurrence of an object.

RcKeyValueDictionary

For RcKeyValueDictionary, reduced conflict operation is achieved by recording adds and removes in a RedoLog. If a transaction conflict is encountered, objects within the dictionary are selectively aborted and the add and remove operations are replayed and a second commit is attempted. You can avoid conflicts under the following conditions:

• Any number of sessions read values in the dictionary at the same time.
• Any number of sessions add keys and values to the dictionary at the same time, unless a session tries to add a key that already exists.
• Any number of sessions remove keys from the dictionary at the same time, unless more than one session tries to remove the same key at the same time.
• Any number of users perform any combination of these operations.

Reduced Conflicts for Indexed Collections

In a future GemStone 101 article, I will go into a little more detail about indexed collections, suffice to say that if you need ordered access to, or need to perform queries on elements in a shared collection, you can use an RcIdentityBag with an RC equality index or two to reduce the likelihood of transaction conflicts.

Concurrency Control in Seaside

In GLASS we have modified the Seaside framework to use both object locks and reduced conflict classes to avoid transaction conflicts.

As I have written before, when a request comes into a GLASS vm, the framework does a beginTransaction before passing the HTTP request to the Seaside application code and the framework does a commitTransaction before passing the HTTP response to the user’s web browser.

Before we do the beginTransaction, we acquire a write lock on the WASession instance to ensure that no two vms are handling a request for the same WASession concurrently. The write lock is then released when the commitTransaction is performed. The object lock is the correct construct to use here, because the intent is to block other sessions from running concurrently. As an intended side effect, this also means that all session-specific data is protected from transaction conflicts.

Each WAApplication maintains a couple of handler dictionaries that are protected by a Semaphore in the standard Seaside code base. For GLASS, we use instances of the reduced conflict dictionary class RcKeyValueDictionary. In this case it isn’t necessary to completely block access to the dictionaries while they are updated. It is sufficient to ensure that the dictionaries are logically consistent at the end of the transaction - reduced conflict classes are the right answer in this case.

For the data structures (that aren’t already protected by the session lock) in your Seaside application, you can apply these tests to help decide what technique to use to avoid transaction conflicts:

• use write locks when concurrent updates to a data structure cannot be tolerated
• use reduced conflict classes when your usage pattern conforms to the restrictions imposed by the particular reduced conflict class

According to Andres Valloud, the Smalltalk Conference of Argentina appears to have been a success - there is already talk of a Smalltalks 2008. This is very good news for Smalltalk!

Judging from this picture by Edgar J. De Cleene (posted on Squeak-Dev mailing list) it was a packed house at James Foster’s “Getting Started with GLASS” workshop on day 3. Over 40 people attended the workshop, sharing 20 computers running the Squeak-based tools, and all doing work with GemStone/S running in a GLASS appliance on James’ 64bit laptop.

James Foster is spanning the globe with a full complement of GLASS presentations in the next couple of months.

In December, James is putting on a GemStone and Seaside workshop at the Smalltalk Conference of Argentina in Buenos Aires, Argentina - December 10-12, 2007.

In January, James and Monty Williams will be representing GemStone at OOP 2008 in Munich, Germany - January 21-25, 2008.

Then in February James will hit a trio of STUGs on the East Coast:

• Toronto Smalltalk User Group - February 4, 2008
• Ottawa Carleton Smalltalk Users Group - February 5, 2008
• New York City Smalltalk Users Group - February 6, 2008

Mark your calendar and make plans to visit with James when he comes calling in your neighborhood.

James Foster of GemStone will be going to the First Smalltalk Conference of Argentina (December 10, 11 and 12, 2007). The conference is being held in Buenos Aires at the University of Buenos Aires. As of November 14th, there are over 170 registrations.

James will lead two half-day workshops on Wednesday, December 12. The morning session will be an “Introduction to GLASS: Seaside with Transparent Persistence.” The afternoon session will be “Scaling GLASS: Growing Your System.”

If you are ready to Tango with GLASS, then you should head on down to sunny Buenos Aires!

Many of you have heard that there are GemStone/S applications running in production with thousands of vms and that other applications are doing thousands of commits per second. I bet you have been wondering what kind of performance you could get running Seaside on GemStone/S - I certainly was:)

Seriously, I spent most of the summer working on tools and with the exception of a few small tests, I didn’t take the time to focus on performance. When I came back from ESUG, I decided to run some scaling tests.

[For those of you who want to see the results and move on with your surfing, here are a couple of links: Goals, Results, Conclusions].

Background

From the tests that I had run over the summer it appeared that Apache was a limiting factor when trying to run at rates above 30 requests per second. I’d also seen some anomalies in the i/o on Linux, were we’d get flat spots in our performance graphs that appeared to be related to file system buffer flushing. If you have read the post on transactions, then you know that we write tranlog records on every commit (page request), so disk i/o can be a limiting factor.

With the help of our IS guy, it turned out that to get the best performance from Apache, we needed to use the MPM worker module. With the MPM worker module turned on, Apache performance fell way off the radar.

The issue with the i/o anomalies that we observed in Linux has not been as easy to resolve. I spent some time tuning GemStone/S to make sure that GemStone/S wasn’t the source of the anomaly. Finally our IS guy was able to reproduce the anomaly and he ran into a few other folks on the net that have observed similar anomalies.

At this writing we haven’t found a solution to the anomaly, but we are pretty optimistic that it is resolvable. We’ve seen different versions of Linux running on similar hardware that doesn’t show the anomaly, so it is either a function of the kernel version or the settings of some of the kernel parameters. As soon as we figure it out we’ll let you know.

For the purposes of these performance tests, I was able to work around the i/o anomaly by putting extents on raw partitions. In nearly all of the tests, the Shared Page Cache (SPC) is sized large enough to hold the entire working set for the test. Consequently there was very little read activity and the system was able to write dirty pages from the SPC fast enough so that random i/o to the raw extent partitions didn’t affect test results.

Goals

I had three goals in mind when I ran this set of tests.

1. demonstrate anticipated performance of the GLASS appliance.
2. demonstrate production performance for the Web Edition.
3. demonstrate performance potential beyond the Web Edition.

For the GLASS applicance tests I wanted to illustrate what you could expect if you installed the VMWare image on a machine without paying particular attention to disk configurations. To simulate the GLASS applicance, I simply ran the tests using file-based tranlogs.

For the production-scale performance of the Web Edition, I wanted to illustrate what you could expect if you paid attention to the disk configuration (i.e., created some raw partitions and had a box with a minimum of 4 disk spindles).

I also wanted to illustrate what kind of performance improvements you could see if you were to add additional SPC or increase the number CPUs available.

Finally, based on a comment where Ramon Leon suggested I include some ’speed comparisons between GemStone and Squeak’, I’ve included runs against a Squeak vm.

Test Strategy

I decided to base the performance tests on the Seaside Counter example. Since the Counter has dead-simple render logic and no significant application state, it is the perfect application for measuring baseline Seaside performance. For GemStone that means that when running tests against the Counter, we’ll be getting performance numbers that include the overhead of persisting and sharing session state (about 250 objects or 50k bytes per request).

Over the course of the summer I ran across siege, “an http regression testing and benchmarking utility” that is very easy to use. It can smack the heck out of a Web Application without putting too much of a load on the system running the test. It also provides some very basic stats about how your app withstood the barrage. Response Time, Transaction rate, and Concurrency being the most interesting.

Siege basically arranges to fire a number of concurrent requests at a given URL. In benchmark mode, as soon as a response is recieved another request is launched in its place. In this mode you can force an application to its knees - very nice for finding bottlenecks. In internet mode, siege waits a random amount of time before firing off the follow-on request, making for a simulation of what the end users of your application may experience.

I ran my tests using the URL ‘http://penny:8000/seaside/examples/counter’, which, as many of you Seasiders out there will recognize, means that a new Seaside session is created on each hit. For benchmarking purposes, that’s just fine - a little extra load never hurts. In the future, I plan to run some scaling tests that measures intra-session performance.

For quite a while now, I have felt that it was important for folks to plan on running multiple vms when they go into production using GLASS. For the best overall response times, one should plan on running at least one vm per concurrent request, which in practice should be 10 or more. For the benchmark tests, the Counter application is cpu bound, so when siege goes about slamming the web server into the ground the cpus are pegged and cpu contention between the processes becomes a factor. It turns out that with 10 vms running on a single cpu, there is a whole lot of contention going on. So after playing around a bit, I settled on using 5 vms for the benchmark tests while running with 10 concurrent requests. This combo gave good performance numbers in the single core tests while in the 4 core test when all 4 cpus were redlined we would minimize the amount of contention. I also ran a couple of internet tests using a single CPU and 20 vms to confirm that in the wild you could afford to run with more than 5 vms without suffering a performance hit. Finally I ran a benchmark test with 100 concurrent requests to see how the system behaved when it was being slashdotted.

Test Setup

For the hardware I used 3 machines foos, toronto, and penny.

Penny is a 2.6Ghz AMD Opteron, with 2 dual core cpus, running SUSE Enterprise 10, with 8Gb of ram. Penny was used to host Apache and Siege. We had a dedicated 1Gbs ethernet connection between penny and toronto.

Foos is a 2.4Ghz Intel, with 1 dual core cpu, running SUSE Enterprise 10, with 2Gb of ram and 2 disk drives (no raw partitions). Foos was used to simulate the GLASS appliance performance.

Toronto is a 2.2Ghz AMD Opteron with 2 dual core cpus, running SUSE Enterprise 10, with 8Gb of ram and 5 disk drives (raw partitions installed on 3 of the partitions). Toronto was used to simulate a typical production machine.

Siege was pointed at the Apache instance listening on port 8000. In addition to the mpm_worker_module, mod_proxy_balancer was used to round-robin requests to the various vms. GemStone was running with version Seaside2.8g1-dkh.490 of Seaside.

For the Squeak tests I used the latest development image from Damien Cassou (sq3.9-7067dev07.10.1), the 3.9 vm and loaded Seaside2.8a1-lr.492. I pointed siege directly at the Squeak image (both running on Toronto).

I did not enforce exclusive use on any of the machines (penny and toronto are shared by other folks in the company) during the tests. But between running most of the tests multiple times and keeping an eye out for anomalous events the numbers are good enough for government work.

Summary of Results

In all, I ran 15 different tests in 6 different categories (Squeak baseline, Squeak internet, Squeak benchmark, GemStone Web Edition, GemStone internet, and GemStone benchmark).

The following table is sorted by Req/Sec. Click on a Run number to jump to the category and a description of the test.

ART in the Notes column is shorthand for Average Response Time, a stat from siege.

Squeak Results

Run1, Run3, and Run8 are scaling tests against the Squeak image. Run7 and Run12 are comparable tests run against a GemStone vm. Note that there is only 1 GemStone vm being used in these tests.

Baseline

Run8 showed the best results for Squeak with a rate of 32 request/second when hit with a siege from a single user.

Internet tests

For the internet test (Run3), the Squeak vm hit 16 requests/second (0.34 seconds average response time and an average of 6 concurrent requests). In a comparable test (Run7), the GemStone vm hit 29 requests/second (0.02 seconds average response time and an average of 0.6 concurrent requests).

Benchmark tests

When 10 concurrent users slammed the Squeak vm (Run1), performance dropped to 10 requests/second. Siege doesn’t collect stats on the standard deviation, but I observed a wide range of response times around the 1 second average reponse time. In the comparable GemStone test (Run12), the GemStone vm hit 91 requests/second, a standard deviation of 6 and an average response time of 0.1 seconds.

Under load it appears that GemStone is about 10 times faster processing Seaside requests than Squeak (Run1 comparied to Run12). While the GemStone vm is certainly faster than the Squeak vm, I don’t think that the GemStone vm is that much faster. I haven’t tried to analyze what might be going on, but my best guess is that under load, the garbage collector is siphoning cpu cycles away from the processing of requests.

GemStone Results

Web Edition tests

Run2, Run4, and Run9 are intended to illustrate the kind of performance you might expect when running a version of the Web Edition (i.e., 1 core, 1G SPC, and a 4G extent).

Run2 and Run4 used file-based tranlogs and extents and Run9 used raw tranlogs. You can see that Run9 is over 3 times faster than Run2. Using raw I/O for tranlogs makes a big difference.

Even with 2 cores, Run4 is still slower than Run9. Confirming that with file-based tranlogs, the test is i/o bound.

A sustained rate of 15 requests/second (24×7) is about the top rate that we’d recommend when using the Web Edition. In Run2 the disk-based garbage collector kept pace with the expiration of session state consuming roughly 1/5 of the 4G repository before it was garbage collected. During Run10 (at a rate of 75 requests/second) a 4G extent was consumed in 15 minutes!

Internet tests

Run5 and Run6 illustrate what you might expect for production performance with real world loads.

The difference between Run5 and Run6 is that I used 2 cores and a 5G SPC in Run6. It is clear that neither a larger SPC or more cores are needed to sustain this rate.

These tests were run using 20 vms. In GemStone/S a vm handles a single http request at a time (each http request coming into the Seaside vm acquires the transaction mutex for the duration of a request), so in order to get concurrent handling of requests you need to have multiple vms running. A quick look at Run7, which runs at the same rate with a single vm, shows that you don’t sacrifice performance by spreading the load across 20 vms, while gaining the ability to handle up to 20 requests concurrently.

Benchmark tests

Run10, Run11, Run13, Run14, and Run15 are intended to illustrate how GemStone/S stands up to siege in benchmark mode and to illustrate the scaling performance as you add cores into the mix.

Run10, Run13, Run14, and Run15 show a nearly linear progression in performace as more cores are added.

With Run11 I cranked siege up to 100 concurrent requests to see what would happen. The fact that Run11 averaged a little more than Run10 even thought they are running on the same configuration, means to me that the cpu wasn’t running flat out in Run10. With 100 concurrent requests, though, the cpu was truly hammered. If you look at the Average Response Time for the two runs, you’ll see that with 10 times more concurrent requests, it took 10 times longer to respond to a request, which makes sense since the extra concurrent requests ended up being queued up behind the 5 vms.

At rates approaching 200 requests/second, I started seeing indications that the data structure used to store session state (RcKeyValueDictionary) was reaching the limit of its effectiveness. As we move forward I will be looking at data structures that are better suited to these rates.

Conclusions

You can expect very reasonable performance numbers with the Web Edition - pretty good value for your money:). Along with transparent persistence, the Web Edition will support rates up to 50 requests/second across a large number of vms. You will have to keep an eye on the size of your repository, if your sustained rate starts averaging near 15 requests/second. But hey, you can serve an awful lot of donuts at these rates.

If you start getting more traffic, it looks like GemStone/S can be scaled to some pretty respectable rates without having to change your application.

Check out a podcast that Monty Williams, James Foster and I had with the folks from Cincom (Michael, and Arden, Dave Buck and James Robertson) where we talked about GLASS and Seaside.

This afternoon Mike Culbertson (our IS guy) and I got a version of GemSource running with lighttpd, with no changes required on the GemStone side (way to go James). The interesting thing here is that lighttpd comes with FastCGI built-in (no compiling as required for mod_fastcgi in Apache) and lighttpd automatically load balances the FastCGI requests.

Here’s the config file we used (no load balancing needed with GemSource at the moment). ss is the GemSource application and config is the Configuration Editor (requiring authorization):

server.modules   += ( "mod_fastcgi" )
fastcgi.server    = (
"/ss" =>
  ((
  "host" => "10.80.250.190",
  "port" => 9765,
  "check-local" => "disable",
  "mode" => "responder",
  )),
"/config" =>
  ((
  "host" => "10.80.250.190",
  "port" => 9765,
  "check-local" => "disable",
  "docroot" => "/htdocs",
  "mode" => "authorizer",
  ))

)

Another webserver option for GLASS. Does that mean we’ll have to call it GLlSS, or maybe GLLASS?

With this post I am starting a series of articles on some of the fundamental concepts that you should have at least a passing familiarity with if you are planning on working with GemStone/S. These articles aren’t intended to replace our most excellent documentation, but to serve as a tantalizing introduction to the wonderful delicacies one can enjoy if he or she decides to crack the shell of a fresh SAG(pdf), open a case of Topaz(pdf) and sit by a cheery fire some chilly evening this autumn.

First on the menu is the transaction. GemStone/S transactions are light on the nose and have a warm, smokey flavor with echoes of strawberries and glazed donuts that reverberates off the palate with the force of a Dandelion archene….ahem…one shouldn’t write a blog on an empty stomach.

GemStone/S is a full-fledged database complete with ACID properties. Not ACID as in Lucy in the Sky with Diamonds, but ACID as in a guarantee that the Smalltalk objects in your transaction have been completely and correctly written to disk: Atomicity, Consistency, Isolation, and Durability.

In GemStone/S transactions are used for persistence, but they are also useful for sharing objects between vms, via the Shared Page Cache (SPC).

I will warn you that I’ll get a little geeky during the following discussion and it will probably help if you take a peek at the Glossary to familiarize yourself with some of the terms I’ll be using, but I promise to try to avoid dragging you too far down the rabbit hole.

Abort

All transactions in GemStone start with an abort. When an abort is performed the following steps are performed in a gem:

1. Invalidate objects
2. Acquire new view

Invalidate objects

All of the dirty objects (i.e., persistent objects previously modified) in the vm are marked as invalid. The list of dirty objects in a vm is called a writeSet. All objects that have been changed since the last time the gem updated its view (called a writeSetUnion) are also marked as invalid.

A subsequent reference to an invalid object will cause a fresh version of the object to be copied into the vm.

Each persistent object is identified by a unique id called an OOP. An OOP is a 61 bit value. The three extra bits in a 64 bit word are used as tag bits. Tag bits differentiate between regular objects (which need to be physically stored in the data base) and special objects. The value of a special object is encoded in the 61 bits of the OOP itself. SmallIntegers, SmallDoubles, and Characters are examples of special objects.….Was that a rabbit????

One or more objects are stored on a data page. Objects that are larger than a page are transparently broken up into page-sized chunks. Because of this chunking it is possible to reference objects in a million element array without having to load the entire million element array into memory.

Objects are read from and written to extents on disk in units of pages. Not suprisingly pages are cached in memory in the Shared Page Cache.

The Object Table (OT) is a Btree that maps OOPs to pages. The OT is stored in a set of pages, cached in the SPC and written to an extent just like the data pages.

Acquire new view

The gem contacts the stone and gets a new view of the database. A view is a reference to the latest OT along with some other bookkeeping information.

Transaction Body

As the vm executes code following an abort, it starts keeping track of all of the objects that are modified during the transaction in the writeSet.

Object references are stored in the body of an object as an OOP, so when an instance variable is accessed in a persistent object and the object is not in the vm or the object has been invalidated, the OT is consulted, and the SPC is checked to see if the page containing the object is already present. If the page isn’t in the SPC, then the page is loaded from disk. Finally, the object is copied from the page into the vm.….A white rabbit????

Commit

When a commit is performed the following steps are performed in the gem:

1. Flush dirty objects
2. Write transaction log entries
3. Acquire commit token
4. Check for conflicts
5. Finalize commit

Flush dirty objects

During this step all of the objects in the writeSet and all of the newly created objects that are reachable from persistent, dirty objects are copied from the vm into new pages in the SPC.

The gem’s copy of the OT is then updated with the new OOP to page mapping. The OT data structure is designed to do a copy on write, so only the portion of the Btree that is changed needs to be written to new OT pages. In practice large portions of the OT are shared amongst multiple views.….Nope, it’s a Dormouse. I’m sure of it.

The new pages in the SPC, containing the latest state of the modified objects are not written directly to disk as part of the transaction. Doing so would take too much time, as disk writes are notoriously slow. A separate process (AIO pageServer) asynchronously writes the ‘dirty’ data pages to disk.

At periodic intervals a concerted effort is made to ensure that all ‘dirty’ pages written before a certain point in time are flushed to disk. This is called a checkpoint.

Write transaction log entries

In order to ensure Durability we do have to write something to disk as part of the transaction. It turns out that we can write a minimum amount of information about the changes to objects much faster than we can write the entire object to disk.

During this step, tranlog records are written by the stone for all of the changed and new objects. Asynchronous i/o is used (when available on the host os) to write transaction logs, so that commit processing can go on while the tranlog records make their way to disk. In a performance sensitive installation, the transaction logs are located on a raw partition or on optimized disk arrays for the fastest i/o possible.….Oh Oh, now there’s a wacky, little guy in a top hat.

In the event of a system crash, one can recover the database by replaying all tranlog records written since the last checkpoint.

Acquire commit token

Up to this point commit processing in multiple gems can occur in parallel, but in this final phase of the commit, only one gem can proceed at a time. The stone manages the queue of gems by handing out a commit token to one gem at a time.

Check for conflicts

When a gem gets the commit token, it begins checking for commit conflicts (i.e., a valid transaction). It does this by comparing the gem’s current writeSet with the writeSetUnion (the union of all writeSets from the transactions that occurred since the gem acquired its original view) and if any of the OOPs are in both sets a conflict has occurred and the commit fails. If the commit fails, the gem gives up the commit token and either aborts or attempts to recover from the commit failure.….Little balls of ….. hedgehogs????

Finalize commit

If there are no conflicts, then the gem returns the commit token to the stone along with a copy of its writeSet (for writeSetUnion processing in other gems) and a reference to the gem’s updated OT. The stone is then free to pass along the commit token, but the gem must still wait until the stone informs it that the asynchronous transaction log i/o has completed.

….a toothy smile appears floating in mid-air and the Cheshire Cat slowly materializes, hands you a precious stone and fades away completely….

Oh well, maybe I went farther down the rabbit hole than I originally planned, but I did warn you!

There’s an interesting story in Waters about Kapital(pdf), a financial risk management and pricing system that was implemented in the early 90’s using VisualWorks and GemStone/S at JPMorgan. The system has been in use for over 13 years at JPMorgan and will continue to be used ‘for the foreseeable future’, largely because the system was written in Smalltalk and is giving the developers at JPMorgan the ability to stay ahead of the competition.

As a Gemstoner, I like the following quote (emphasis mine):

A small PCS installation of a few hundred CPUs will remain to support the Smalltalk GemStone database and those processes that are closely linked to it. “This will maintain the flexibility that developers have found so valuable,” says Verdier. “It is the best of both worlds.”

Check it out.

The folks at Finworks have announced that their Group Investment System developed for Allan Gray went into production yesterday (September 27). The application was written in Seaside and runs on top of GLASS. Finworks was our first Beta customer and as I’ve reported in a number of posts were actively contributing to the tools work over the summer (here, here, here and here) while developing their application.

Head on over to the Seaside Users page for a little more info about the Group Investment System application and a list of some of the other commercial applications written in Seaside.

At ESUG, Lukas, Philippe and I started porting Magritte and Pier to GemStone and we discovered that the Morphic-based Monticello tools are way too slow. The combination of lots of package versions and a slow wireless connection exposed the weakness in the Morphic-based tools: way too many round trips between the Squeak and GemStone. We’ll have to have OmniBrowser-based Monticello tools before we can widely distribute the G.L.A.S.S. beta.

Fortunately, Damien Cassou as part of the Summer of Squeak project has made an excellent start, implementing an OmniBrowser-based Monticello Browser for Monticello 2. I should be able to adapt his Monticello Browser to Monticello 1 and then use it as a jumping off point for creating the rest of the Monticello tools in OmniBrowser.

I am finally beginning to see the light at the end of the tunnel.

At ESUG, we announced that G.L.A.S.S. will be distributed as a VMWare appliance:

• GemStone/S 64
• Linux
• Apache
• Seaside
• Smalltalk

By bundling everything into a VMWare appliance, we not only eliminate the hassles involved in getting all of the different pieces of software installed and configured to work together, but we also make it possible to run G.L.A.S.S. on any 64 bit hardware, independent of OS. VMWare provides free virtual machines for Windows and Linux. VMWare Fusion is available for Mac OS X.

When the appliance is booted, GemStone/S and Apache are automatically started giving you immediate access to your Seaside applications. A Squeak development image, pre-configured with all of the Monticello packages needed to run the OmniBrowser-based tools is included in the appliance. The development image can be run from within the appliance, or it can be downloaded to your favorite development machine and connected to the GemStone repository for remote development.

Everything you need for deploying a Seaside application in a nice, neat little package.

We’re in limited alpha public beta for the appliance and the tools. If you are interested in participating in the beta program, download the latest version of the appliance and join our Beta mailing listgo to our G.L.A.S.S. site and sign up.

I am satisfied. I’m not done (not by a longshot), but I am satisfied.

In the last couple of (long) days I’ve been able to touch just about all of the tools and while some of them have interesting quirks and foibles, one can get real work done using them. So with many thanks to Liliana Ivan we can stick a fork in the Monticello tools.

The Monticello tools are not based on OmniBrowser so we had to create proxies for big chunks of the Monticello model (read lots of classes) which not only complicated the development process, but also results in more roundtrips between Squeak and GemStone while using the tools. I have not addressed dependency issues either, so there are a handful of update-related problems outstanding. But hey, what’s a couple of bugs amongst friends?

Early tomorrow morning we will begin our trip to ESUG in beautiful Lugano, Switzerland. The good news is that I will be able to sleep on the plane, rather than fix quirks in the tools. The bad news is that I’ll be trying to sleep on a plane.

While we’re in Europe, we’ll release a GLASS alpha to some of the folks at ESUG. They can kick the tires, take the tools for a spin and If they take it slow and stay in the middle of the road, they’ll have a nice experience. However, I understand that there are some curvy roads high in the Swiss Alps, so I’m anticipating that they will find a foible or two that I haven’t seen.

After ESUG, my wife and I will spend a few days in Venice, before heading back to the states. That’s when we anticipate that a few more folks will be given access to the GLASS alpha.

I grew up in Indiana and every summer we’d hook the old camping trailer up to the family station wagon and head out on the road for a couple of weeks. Over the years we fished for Northern Pike in the far north of Ontario, visited cousins in Anaheim and toured the cliff-side homes of the Ancient Pueblo. Most summers we’d stay closer to home camping at places like The Dunes, Indian Lake or Brown County.

One year we traveled east to see the sites. We visited Gettysburg, Jamestown and The Smithsonian. I have vivid memories of the hex signs on the barns in Pennsylvania Dutch country, Maryland crab cakes, the lush Shenandoah Valley and a dusty, old Civil War museum in Appomattox. When we visited Monticello, I remember learning that in those days, the kitchens were always built separate from the house, because of the danger of fire. I remember thinking that the soup must get cold when they carry it from the kitchen to the dining room.

Colin might claim that he was thinking about another place when he named Monticello and my mom might claim that we visited Mount Vernon not Monticello that summer, but these are my memories, my story and no one can deny the fact that whenever I work on the Monticello tools my mind wanders back to that summer some 40 years ago.

I spent most of the last week getting the Debugger and Inspector into shape. While I’m still not completely happy with the Debugger, it is definitely usable. Every once in a while though, it lets you down.

Yesterday I started to make a pass through the Monticello tools. I’ve been using the Monticello tools full-time for the last week or so, but I need to make sure that the functions I don’t use on a regular basis are ready.

With the way things are progressing, we should have something interesting to show at ESUG.

Man those crab cakes were good!

After getting the OmniBrowser tools running against Gemstone/S last week, I started working on the Debugger and Inspector.

Fortunately, Lukas Renggli had done an OmniBrowser-based Debugger and Inspector and I was able to port it to GemStone/S in relatively short order, leveraging James Foster’s previous work on a Morphic-based debugger. The heavy lifting involved getting the Squeak-side process management under control and generalizing the intervm notification mechanism to use ClientForwarders - taking a page from GemBuilder for Smalltalk.

To top it all off, Liliana Ivan (from FinWorks) has managed to bring the Monticello tools to the point where we can load, save, and view changes!

We are getting danged close.

Now that all of the tools are functional, we’ve only got 80% of the work left. Ha ha!

Seriously, I still have to make an exhaustive pass through the Monticello tools (Liliana has taken a vacation of all things), Inspector and Debugger to flush out any fundamental problems.There are also a handful of annoying update (or lack thereof) problems that have to be ironed out before the tools can be given a wider audience. All in all, we are still on a pace to open things up before ESUG.

If you like your tools medium rare, that is.

Yesterday I declared a cage match with OmniBrowser - I wasn’t leaving until I had a functional set of OmniBrowser tools. I made a sweep through all of the menus and windows reachable from OBSystemBrowser, fixing problems or lopping off features along the way. It took a while, but in the end the OmniBrowser tools are ready for prime time.

Strarting today, I will do all of my development activities for GLASS using the OmniBrowser-based tools.

Next up is to tackle the debugger and inspector. Before we can open up the beta to a wider audience, we need a useable debugger and inspector - I can get by with what we have, but I wouldn’t ask anyone else to try.

Liliana Ivan is continuing to make progress on the Monticello tools, so I am optimistic that we’ll be able to open up the beta to a wider audience before ESUG, which would be nice.

Since last week we’ve made some very good progress on the tools. As I was hoping, once the OBBrowseRequest was hooked up, I was able to open all of the OmniBrowser tools on the Squeak-side. There’s a single Squeak-side class, GsOBBrowser, that proxies for all of the tools classes (subclasses of OBBrowser). Now it’s a matter of chasing down the bugs in GemStone-side OG-Standard classes (a port of the OB-Standard tools package) - there are a couple of outstanding issues, but at this point it is all about finding the time to turn the crank.

The tools are in good enough shape I have started to use them to browse the code, as I do other Seaside-related development.

I did get fed up with how long it was taking to generate the browser menus (one round-trip per menu item, yikes!), so I had to nail that one, pronto. Now it takes only one round-trip, with no noticeable delay. This is real encouraging, I had been concerned about performance, but I think we should be able to make most of the operations acceptably fast.

Liliana Ivan from Finworks started contributing to the Monticello tools this week. We’re adapting the existing Monticello tools instead of doing OmniBrowse-based tools (building on the initial work done by James Foster). I think this is the fastest route to a complete tool set, we’ll just have to backfill with OmniBrowser tools later.

As I reported last week, Seaside 2.8 has been ported to GemStone/S. It turns out that Scriptaculous (Scriptaculous-lr.206.mcz) runs on GemStone/S nearly out of the box - adding 2 ‘missing’ Squeak methods and a change to one of the Scriptaculous methods were all that was needed to get it running.

This week (okay, Tuesday) we ported SeasideTesting to GemStone/S to run on top of Seaside 2.8. The GemStone/S version of Seaside2.8 was branched from Seaside2.8g1-pmm.405. When the tests are run, we get the same set of errors in GemStone/S that you would get running the tests in Squeak against 2.8 - one extra error, because GemStone is missing the #years method in DateAndTime (another ‘missing’ Squeak method).

After my vacation, I’ve been able to spend time focusing on the tools and I’m making good progress. The OmniBrowser-based System Browser is coming alive - it’s about three quarters functional, with most of the hard problems solved. Once the SystemBrowser is fully functional, the rest of the OmniBrowser tools (HierarchyBrowser, ListBrowsers, etc.) should fall in short order.

The approach I’m taking is to create Squeak-side proxies for the OBSwitch, OBPanel and OBColumn classes, so there is very little browser-specific code that needs to be written in Squeak. The Morphic code deals almost exclusively with instances of these classes - there has been a little monkey-business with menus, but they are falling nicely into line.

If you want to keep an eye on my progress you can browse around in the GemStone Tools project over on GemSource. We’ve still got the debugger, inspector, and Monticello tools to finish before we can open the beta up to more users.

With some help from the folks at FinWorks and the groundwork laid by Philippe Marschall, Lukas Renggli and the rest of the Seaside crew, we’ve got Seaside2.8 ported to GemStone/S.

I’m actually on vacation this week (haha), which is a testament to how easy it was to port 2.8 to GemStone/S. Good work guys (and gals)!