2008 GemStone/S Training Courses Announced

GemStone is pleased to announce the availability of the following open enrollment courses:

1. GemStone/S Fundamentals – October 20 – 24, 2008. Cost: $2300
2. GemStone/S Performance – October 27 – 29. Cost: $2300
3. GemStone/S Advanced Programming Practices – November 3 – 7, 2008. Cost: $3300

All courses will be presented at GemStone Headquarters in Beaverton, Oregon. Course descriptions are available upon request. Students will also have an opportunity to meet the GemStone Engineering team and ask questions about any and all things GemStone/S.

GemStone reserves the right to cancel any course due to insufficient enrollment. At least 14 days notice will be given in case of cancellation in most cases. Minimum enrollment is 3 students per course.

Please send enrollment requests, including student’s name, desired course(s), and form of payment to: training@gemstone.com.

Class details can be found here.

GemStone/S Training – Fall 2008

Plans are firming up for a training class to be held in Portland, Oregon in October. If you’d like to attend the class, send mail and Norm Green will get back to you.

Class details can be found here.

GLASS on Tour

James Foster will be hitting the road again this fall. This time around, James will be on a whirlwind tour of the Midwest. Right now he’s working on plans to talk to the folks at the Cincinnati Agile Round Table about Seaside and GLASS.

If you’d like James to pay a visit to your group, send mail and he will get back to you.

GemStone/S 64 2.3 Beta 2 is available

Head on over to the GLASS downloads page for the latest drop of the appliance (version 1.0beta10). The downloads for Linux and Mac OS X Leopard have been updated as well.

The big news for 2.3 beta2 is that we’ve got support for the full Unicode character range. Previously we supported up to 16 bit characters with DoubleByteString. In 2.3, we’ve added the class QuadByteString for supporting up to 32 bit characters. There should be more information in the release notes for 2.3, but until the release notes are available, you can review the chapter on “Extended Character Set Support” in the 2.2 Release Notes.

The following bugs were fixed:

• bug39137 – ‘Magritte validation error (specifically for MARequiredError)’
• bug39246 – ‘URL can cause fastcgi infinite recursion and vm crash’
• bug39274 – ‘RRRssHandler>>handleRequest: ObjectLog support for debugging problems’

The appliance also includes some performance improvements for FastCGI.

Beta 2 is shipped with GLASS.230-dkh.155 and GemStone-jgf.291(both can be found in the GLASS project on GemSource). If you are using an external Squeak image, you should probably use the image and changes files that are shipped with the release:

• You can download the GemStone ‘GemTools one-click’ from http://seaside.gemstone.com/downloads.html [thanks Monty!].
• For the appliance you can download the GemStone one-click from the appliance itself by navigating to the downloads page in your browser (http://<appliance ip address>/downloads.html).
• For the standard (Linux or Mac) install, the image and changes can be found in $GEMSTONE/seaside/squeak_3.9 and the latest GCI library $GEMSTONE/lib/libgcsi64-230.so.

If you are using 1.0beta9, I encourage you to download and start using the new version of the appliance.

With ESUG around the corner, I have not had time to merge the bugfixes into GLASS-dkh.119 (for 2.2.5). I hope to publish a beta update during the conference or shortly afterwards.

1 Session per VM: Another Scaling Alternative

I know, I know, at first blush it sounds like a bad idea, but if you let the idea marinate overnight and then sear over red hot mesquite – it ends up being a pretty tasty idea. No that isn’t the sound of vertebrae popping in the background:)

I blame Ryan Simmons. In a comment to my previous post, he innocently asked the question (emphasis mine):

Would it be possible to not comit session state to the stone and use something like Ramons current post on scalling ( http://onsmalltalk.com/programming/smalltalk/seaside/scaling-seaside-more-advanced-load-balancing-and-publishing/ ) to redirect users to the same gem.

Avi Bryant read Ryan’s comment and suggested that running Seaside on GLASS using 1 Session per VM could be a viable technique for avoiding commits.

But, 1 Session per VM?

As I’ve detailed elsewhere, there are good reasons for not using a vm to serve multiple concurrent sessions (without doing a commit per request), but if one were to serve a single session per vm, the good reasons are rendered moot.

With 1 Session per VM, session state does not need to be persisted and voila! no commits due to changes in session state. One should be able to approach the same sort of performance achieved with navigation URLs (i.e., 130 pages/second running on commodity hardware: 2.4Ghz Intel, with 1 dual core cpu, running SUSE Enterprise 10, with 2Gb of ram and 2 disk drives – no raw partitions) while continuing to use rich, stateful Seaside components.

Isn’t 1 Session per VM wasteful?

It depends upon how you look at it. The only thing that might be wasted is memory/swap space. A couple of hundred extra processes on todays hardware is not a big deal unless you are swapping.

If you have 100,000 unique visitors per month and a 10 minute session expiry, you end up needing around 20 sessions. Throw in a fudge factor of 5x and you’re looking at 100 sessions.

100 VMs at 100Mb per VM (tunable) should consume 10Gb of RAM, unless you use mmap (which we do). With mmap, only the memory that is actually used is allocated in RAM. The GemStone/S object manager maps and unmaps chunks of memory on demand so the full 100Mb will only be used if needed and when the memory is no longer needed, it is returned to the system. Without some real benchmarks I can’t tell for sure, but I think it is reasonable to assume that 100 100Mb GemStone vms could run comfortably in 5Gb or less of real memory.

On the flip side, in-vm garbage collection is much cheaper, because only the working set for a single session needs to be swept. When session state for multiple sessions is colocated in the same vm, there is noticeable overhead, so with 1 Session per VM we trade memory for CPU.

Conclusions

We will run some real benchmarks to characterize the tradeoffs between memory, CPU, and commits, but based on back of the envelope calculations it appears that 1 Session per VM is a viable approach for scaling Seaside applications with GemStone/S.

I’m headed to Amsterdam and ESUG on Thursday, but I intend to get busy on this when I get back into Portland in early September, so keep your eyes peeled.

Making ‘_k’ and ‘_s’ optional: A Seaside Heresy?

“Nobody expects the Spanish Inquisition!” With that Cardinal Ximinez spun on his well-shined heel and pulled the heavy, copper strapped cell door closed. Laughter echoed in the tunnel as the meager light drifted away with the torch smoke.

I didn’t set out to commit heresy, I suppose noone really does.

Sprawled on a bug infested pallet, locked in a cell deep beneath the Abbey, remnants from my former life draped in ragged tatters from my bruised limbs, I wondered if my simple act of heresy was worth the heavy price I am doomed to pay. I hold scant hope for my redemption, but you gentle reader, you have a chance to learn from my descent into dissent.

Back in April…

…after resolving the last couple of issues with transparent persistence, I turned towards the scalability and performance of Seaside and GLASS. For focus I set a hypothetical goal of 100,000 requests/second – if you’re going to aim for something you might as well aim high.

Since GLASS performs one commit per HTTP request I started out by tuning GemStone/S for maximum commit performance. Before long I realized that I was barking up the wrong tree – if our commercial customers can do say 5,000 commits/second, then I’d need to come up with a 20x performance boost for commit processing to reach 100,000 requests(commits)/second – an admirable goal but not very realistic – we’ve spent 20 years tuning commit performance.

For a 20x performance improvement, I’d just have to pack more bang into each commit or somehow avoid state changes altogether. I’d already explored the ‘more bang per commit’ option way back when I first ported Seaside to GemStone, so I knew that there wasn’t going to be much gained by that route. With regards to avoiding commits, the theory is that avoiding commits for 95% of the pages could result in a 20x gain in performance. The more commits avoided the bigger the multiplier. Now we’re talking!

In May, I started exploring the reuse of session state. By the end of July, I had a proof of concept demonstrating the reuse of 99% of the session state, but I wasn’t real happy with the extent of changes I had to make to Seaside. Nor could I eliminate the decoration chain update used with #call:/#answer: the primary navigation scheme for Seaside. We aren’t going to avoid commits on 95% of the pages without being able to navigate statelessly. I need a navigation scheme that doesn’t require Seaside session state.

Last week (end of July) James Foster and I tossed around some ideas for doing navigation in Seaside without using #call:/#answer: and without using Seaside session state. After playing around with a couple of approaches I settled on the following style of URL, which I’ll call a navigation URL – it’s sorta RESTful, but not too RESTful, it is bookmarkable and it was pretty easy write a component to render pages (the most important – Ha, Ha):

http://example.com/sushi?_n&pg=Home
http://example.com/sushi?_n&pg=Sale
http://example.com/sushi?_n&pg=Catalog
http://example.com/sushi?_n&pg=CartPage
http://example.com/sushi?_n&pg=Item&id=26

I wrote an example Seaside application (Sushi-Example) using the navigation URLs. It is available on GemSource in the GemStone Examples project. As of this writing Sushi-Example-dkh.25 is the version that should be used. The Sushi-Example package can be loaded into either GLASS or Squeak Seaside. There is no styling in the example so don’t expect anything pretty. I encourage you to take a look at the example and give us feedback.

Last Sunday morning I was lying in bed mulling over the problem when it occured to me that if I dropped the ‘_k’ off a navigation URL and figured out how to render the page, I wouldn’t need to save a continuation for each page hit and most importantly I’d avoid a commit for each of those pages as well! After a little more thought, I figured it would be possible to avoid saving sesssions making it possible to drop the ‘_s’, too. By Monday afternoon, I had a prototype running.

I spent Tuesday doing some benchmarks to see if there were noticeable improvements in performance. I knew that by avoiding commits I would greatly improve the scalability of Seaside, but I was also hoping to see some performance gains, too.

I have to admit that advocating the use of navigation URLs and proposing the ‘_k’ and ‘_s’ be optional URL parameters makes me feel like a Seaside heretic…

(JARRING CHORD. The door flies open. In come three evil types in red robes.)

Cardinal Ximinez: NOBODY expects the Spanish Inquisition! Our chief weapon is surprise…surprise and fear…fear and surprise…. Our two weapons are fear and surprise… and ruthless efficiency…. Our three weapons are fear, surprise, and ruthless efficiency… and an almost fanatical devotion to the Pope…. Our four… no… Amongst our weapons… Amongst our weaponry… are such elements as fear, surprise… I’ll come in again.

A Case to make ‘_k’ and ‘_s’ optional using Navigation URLs

… but here goes, the rack be damned.

I have glossed over a couple of details in the following section, so you’ll want to look at the code for the real skinny:)

In a normal Seaside application URLs are generated for links that look something like this: http://example.com/sushi?_s=68pqfS&_k=SW7A&_n. To decode this URL Seaside uses the ‘_s=68pqfS‘ to lookup a session, the ‘_k=SW7A‘ to look up a continuation in the session and the rootComponent associated with the continuation is rendered. The ‘_n’ indicates to Seaside that no redirect is needed for the rendering pass.

Consider a navigation URL that looks like this: http://example.com/sushi?pg=Item&id=26. To decode the navigation URL, Seaside creates a default session (no ‘_s’ present) and renders the rootComponent associated with a default rendering continuation (no ‘_k’ present). The rootComponent uses the ‘pg=Item&id=26′ to lookup and render the appropriate item.

When a navigation URL is used to reach a page, Seaside creates and saves a new instance of the session class. If a customer is window shopping (i.e., navigating around the site without logging in or adding an item to a cart), the session state is uninteresting and doesn’t really need to be saved. If a request URL has no ‘_s’ and no state is changed while processing the URL, the ‘_s’ could be dropped from generated navigation URLs with no loss of information. There’d be a definite advantage for web sites with lots of window shoppers as only the interesting sessions would be saved.

If a customer logs in or adds an item to a cart, then the session becomes interesting and it makes sense to propogate the ‘_s’ to all generated navigation URLs. Such an URL looks like this: http://example.com/sushi?pg=Item&id=26&_s=68pqfS&_n. To decode this URL, I would expect Seaside to use the ‘_s=68pqfS‘ to lookup a session and render the ‘pg=Item&id=26′ item. The ‘pg=Item&id=26′ is retained in the URL to make it possible to lookup and render the target page. Essentially the ‘pg=Item&id=26′ replaces the ‘_k’, which ends up serving two useful purposes:

• In a window shopping scenario, the customer will continue to navigate around the site. The interesting session state is encoded in the ‘_s’ and without a ‘_k’ there is no need to create and save session state for each page rendered.
• such a navigation URL is natively bookmarkable. If the session expires before the bookmarked URL is used, it still contains enough information to allow a valid navigation to the bookmarked page.

With navigation URLs, a ‘_k’ is needed only when a #callback: or #call: is associated with a rendered anchor or form.

(Torchlit dungeon. We hear clanging footsteps. Shadows on the grille. The bootsteps stop and keys jangle. The great door creaks open and Ximinez walks in and looks round approvingly. Fang and Biggles enter behind pushing in the dear old lady. They chain her to the wall.)

Ximinez : Ha! Now, old woman! You are accused of heresy on three counts. Heresy by thought, heresy by word, heresy by deed, and heresy by action. …Four counts. Do you confess?

Old Lady : I don’t understand what I’m accused of.

Ximinez : Ha! Then we shall MAKE you understand! Biggles! Fetch… …THE CUSHIONS!

The Benchmark

In the Sushi Example, you can use either the standard WASession or the SushiSession. When you use WASession, you’ll get ‘_s’ and ‘_k’ parameters generated for every link (along with the associated session state), while using the SushiSession, the ‘_s’ and ‘_k’ parameters are only included in the URL when needed. Since I wanted to measure the effect of the accumulation of session state (or lack thereof), I decided to run the tests for 20 minutes (double the default sessionExpory). With a 20 minute test we’ll generate a full complement of session state and presumably settle into a steady state as aged sessions expire.

For all of the tests (except Run5), I used the following siege invocation:

siege -b -d 0 -c 10 -t 20M http://example.com/seaside/examples/sushi

The arguments tell siege to hit the URL as hard as possible simulating 10 concurrent users for 20 minutes. For more on siege, see my post Scaling Seaside with GemStone/S.

For Run5, I had to increase the -c argument in order consume both CPUs on the box. Run5 was made with 300 concurrent sessions.

The Benchmark Results

The following table summarizes the results:

Run	Req/Sec	Core	Gem	VM	Machine	Session Class
1	11	1	1	S	laptop	WASession
2	14	1	3	G	Foos	WASession
3	67	1	1	S	laptop	SushiSession
4	129	1	3	G	Foos	SushiSession
5	335	2	3	G	Foos	SushiSession

Run1 (Squeak) and Run2 (GemStone) are baseline runs and if you look at the results from my benchmarks last fall, you’ll see that the figures are comparable (previous Squeak results and previous GemStone results).

Things get interesting when you look at Run3 (Squeak). Not saving session state results in a 6x improvement in performance. Since both runs (Run1 and Run3) were made against the same Sushi-Example the difference in performance can be attributed to the reduced load on the memory management system. While I didn’t measure the size of the VM, I am sure that it is a lot smaller in Run3.

For GemStone Run4 the improvement is even more dramatic – a 9x improvement. For GemStone/S, the improvement is due to the fact that without saving session state, no commit processing overhead is incurred. I should note that on the machine Foos, no effort was made to use raw tranlogs (see Scaling Seaside with GemStone/S and GemStone 101: Transactions for info about raw tranlogs and performance) so the improvement is more dramatic than you’d see on a system that was tuned for commit performance.

For Run5 (GemStone) I added an adidtional CPU allowing the stone and 3 gems to utilize both CPUs available on the machine. This time the improvement was 2.5x. You’d expect a 2x, so the extra margin of performance is probably due to the fact that the there was very little context switching going on with 2 CPUs.

Conclusions

The quest for 100,000 pages/second doesn’t look quite as quixotic now as it looked a week ago. I think that using navigation URLs with Seaside is a valid technique especially if you are interested in a highly scalable Seaside application.

If you are using the Web Edition of GLASS, using navigation URLs will definitely make it possible to sustain rates in excess of 15 pages/second, since you will be able to avoid saving uninteresing session state in the repository.

For low traffic sites, I think navigation URLs hold promise for making it possible to get the most bang for your resource buck.

The prototype for making the ‘_k’ and ‘_s’ optional parameters is less than a week old and I know that there are some weak spots in the implementation, but I’m sure that they can be resolved:

I don’t like how navigationOnly is used in SushiSession. I’m sure that navigationOnly can be eliminated, but it might be necessary to change the logic for generating Form html, since I added navigationOnly so that Forms would work correctly.

I would like to be able to create Forms where an URL could be used as an alternative to using a #callback:. I know I risk be boiled in oil for this, but I would have liked to include a search field on nearly every page, but that would have required a #callback:.

After spending bits and pieces of 3 months working part time towards the reuse of session state, it is pleasing to see how little code it took to make the ‘_k’ and ‘_s’ parameters optional.