“Nobody expects the Spanish Inquisition!” With that Cardinal Ximinez spun on his well-shined heel and pulled the heavy, copper strapped cell door closed. Laughter echoed in the tunnel as the meager light drifted away with the torch smoke.

I didn’t set out to commit heresy, I suppose noone really does.

Sprawled on a bug infested pallet, locked in a cell deep beneath the Abbey, remnants from my former life draped in ragged tatters from my bruised limbs, I wondered if my simple act of heresy was worth the heavy price I am doomed to pay. I hold scant hope for my redemption, but you gentle reader, you have a chance to learn from my descent into dissent.

Back in April…

…after resolving the last couple of issues with transparent persistence, I turned towards the scalability and performance of Seaside and GLASS. For focus I set a hypothetical goal of 100,000 requests/second – if you’re going to aim for something you might as well aim high.

Since GLASS performs one commit per HTTP request I started out by tuning GemStone/S for maximum commit performance. Before long I realized that I was barking up the wrong tree – if our commercial customers can do say 5,000 commits/second, then I’d need to come up with a 20x performance boost for commit processing to reach 100,000 requests(commits)/second – an admirable goal but not very realistic – we’ve spent 20 years tuning commit performance.

For a 20x performance improvement, I’d just have to pack more bang into each commit or somehow avoid state changes altogether. I’d already explored the ‘more bang per commit’ option way back when I first ported Seaside to GemStone, so I knew that there wasn’t going to be much gained by that route. With regards to avoiding commits, the theory is that avoiding commits for 95% of the pages could result in a 20x gain in performance. The more commits avoided the bigger the multiplier. Now we’re talking!

In May, I started exploring the reuse of session state. By the end of July, I had a proof of concept demonstrating the reuse of 99% of the session state, but I wasn’t real happy with the extent of changes I had to make to Seaside. Nor could I eliminate the decoration chain update used with #call:/#answer: the primary navigation scheme for Seaside. We aren’t going to avoid commits on 95% of the pages without being able to navigate statelessly. I need a navigation scheme that doesn’t require Seaside session state.

Last week (end of July) James Foster and I tossed around some ideas for doing navigation in Seaside without using #call:/#answer: and without using Seaside session state. After playing around with a couple of approaches I settled on the following style of URL, which I’ll call a navigation URL – it’s sorta RESTful, but not too RESTful, it is bookmarkable and it was pretty easy write a component to render pages (the most important – Ha, Ha):

http://example.com/sushi?_n&pg=Home
http://example.com/sushi?_n&pg=Sale
http://example.com/sushi?_n&pg=Catalog
http://example.com/sushi?_n&pg=CartPage
http://example.com/sushi?_n&pg=Item&id=26

I wrote an example Seaside application (Sushi-Example) using the navigation URLs. It is available on GemSource in the GemStone Examples project. As of this writing Sushi-Example-dkh.25 is the version that should be used. The Sushi-Example package can be loaded into either GLASS or Squeak Seaside. There is no styling in the example so don’t expect anything pretty. I encourage you to take a look at the example and give us feedback.

Last Sunday morning I was lying in bed mulling over the problem when it occured to me that if I dropped the ‘_k’ off a navigation URL and figured out how to render the page, I wouldn’t need to save a continuation for each page hit and most importantly I’d avoid a commit for each of those pages as well! After a little more thought, I figured it would be possible to avoid saving sesssions making it possible to drop the ‘_s’, too. By Monday afternoon, I had a prototype running.

I spent Tuesday doing some benchmarks to see if there were noticeable improvements in performance. I knew that by avoiding commits I would greatly improve the scalability of Seaside, but I was also hoping to see some performance gains, too.

I have to admit that advocating the use of navigation URLs and proposing the ‘_k’ and ‘_s’ be optional URL parameters makes me feel like a Seaside heretic…

(JARRING CHORD. The door flies open. In come three evil types in red robes.)

Cardinal Ximinez: NOBODY expects the Spanish Inquisition! Our chief weapon is surprise…surprise and fear…fear and surprise…. Our two weapons are fear and surprise… and ruthless efficiency…. Our three weapons are fear, surprise, and ruthless efficiency… and an almost fanatical devotion to the Pope…. Our four… no… Amongst our weapons… Amongst our weaponry… are such elements as fear, surprise… I’ll come in again.

A Case to make ‘_k’ and ‘_s’ optional using Navigation URLs

… but here goes, the rack be damned.

I have glossed over a couple of details in the following section, so you’ll want to look at the code for the real skinny:)

In a normal Seaside application URLs are generated for links that look something like this: http://example.com/sushi?_s=68pqfS&_k=SW7A&_n. To decode this URL Seaside uses the ‘_s=68pqfS‘ to lookup a session, the ‘_k=SW7A‘ to look up a continuation in the session and the rootComponent associated with the continuation is rendered. The ‘_n’ indicates to Seaside that no redirect is needed for the rendering pass.

Consider a navigation URL that looks like this: http://example.com/sushi?pg=Item&id=26. To decode the navigation URL, Seaside creates a default session (no ‘_s’ present) and renders the rootComponent associated with a default rendering continuation (no ‘_k’ present). The rootComponent uses the ‘pg=Item&id=26′ to lookup and render the appropriate item.

When a navigation URL is used to reach a page, Seaside creates and saves a new instance of the session class. If a customer is window shopping (i.e., navigating around the site without logging in or adding an item to a cart), the session state is uninteresting and doesn’t really need to be saved. If a request URL has no ‘_s’ and no state is changed while processing the URL, the ‘_s’ could be dropped from generated navigation URLs with no loss of information. There’d be a definite advantage for web sites with lots of window shoppers as only the interesting sessions would be saved.

If a customer logs in or adds an item to a cart, then the session becomes interesting and it makes sense to propogate the ‘_s’ to all generated navigation URLs. Such an URL looks like this: http://example.com/sushi?pg=Item&id=26&_s=68pqfS&_n. To decode this URL, I would expect Seaside to use the ‘_s=68pqfS‘ to lookup a session and render the ‘pg=Item&id=26′ item. The ‘pg=Item&id=26′ is retained in the URL to make it possible to lookup and render the target page. Essentially the ‘pg=Item&id=26′ replaces the ‘_k’, which ends up serving two useful purposes:

  • In a window shopping scenario, the customer will continue to navigate around the site. The interesting session state is encoded in the ‘_s’ and without a ‘_k’ there is no need to create and save session state for each page rendered.
  • such a navigation URL is natively bookmarkable. If the session expires before the bookmarked URL is used, it still contains enough information to allow a valid navigation to the bookmarked page.

With navigation URLs, a ‘_k’ is needed only when a #callback: or #call: is associated with a rendered anchor or form.

(Torchlit dungeon. We hear clanging footsteps. Shadows on the grille. The bootsteps stop and keys jangle. The great door creaks open and Ximinez walks in and looks round approvingly. Fang and Biggles enter behind pushing in the dear old lady. They chain her to the wall.)

Ximinez : Ha! Now, old woman! You are accused of heresy on three counts. Heresy by thought, heresy by word, heresy by deed, and heresy by action. …Four counts. Do you confess?

Old Lady : I don’t understand what I’m accused of.

Ximinez : Ha! Then we shall MAKE you understand! Biggles! Fetch… …THE CUSHIONS!

The Benchmark

In the Sushi Example, you can use either the standard WASession or the SushiSession. When you use WASession, you’ll get ‘_s’ and ‘_k’ parameters generated for every link (along with the associated session state), while using the SushiSession, the ‘_s’ and ‘_k’ parameters are only included in the URL when needed. Since I wanted to measure the effect of the accumulation of session state (or lack thereof), I decided to run the tests for 20 minutes (double the default sessionExpory). With a 20 minute test we’ll generate a full complement of session state and presumably settle into a steady state as aged sessions expire.

For all of the tests (except Run5), I used the following siege invocation:

siege -b -d 0 -c 10 -t 20M http://example.com/seaside/examples/sushi

The arguments tell siege to hit the URL as hard as possible simulating 10 concurrent users for 20 minutes. For more on siege, see my post Scaling Seaside with GemStone/S.

For Run5, I had to increase the -c argument in order consume both CPUs on the box. Run5 was made with 300 concurrent sessions.

The Benchmark Results

The following table summarizes the results:

Run Req/Sec Core Gem VM Machine Session Class
1 11 1 1 S laptop WASession
2 14 1 3 G Foos WASession
3 67 1 1 S laptop SushiSession
4 129 1 3 G Foos SushiSession
5 335 2 3 G Foos SushiSession

Run1 (Squeak) and Run2 (GemStone) are baseline runs and if you look at the results from my benchmarks last fall, you’ll see that the figures are comparable (previous Squeak results and previous GemStone results).

Things get interesting when you look at Run3 (Squeak). Not saving session state results in a 6x improvement in performance. Since both runs (Run1 and Run3) were made against the same Sushi-Example the difference in performance can be attributed to the reduced load on the memory management system. While I didn’t measure the size of the VM, I am sure that it is a lot smaller in Run3.

For GemStone Run4 the improvement is even more dramatic – a 9x improvement. For GemStone/S, the improvement is due to the fact that without saving session state, no commit processing overhead is incurred. I should note that on the machine Foos, no effort was made to use raw tranlogs (see Scaling Seaside with GemStone/S and GemStone 101: Transactions for info about raw tranlogs and performance) so the improvement is more dramatic than you’d see on a system that was tuned for commit performance.

For Run5 (GemStone) I added an adidtional CPU allowing the stone and 3 gems to utilize both CPUs available on the machine. This time the improvement was 2.5x. You’d expect a 2x, so the extra margin of performance is probably due to the fact that the there was very little context switching going on with 2 CPUs.

Conclusions

The quest for 100,000 pages/second doesn’t look quite as quixotic now as it looked a week ago. I think that using navigation URLs with Seaside is a valid technique especially if you are interested in a highly scalable Seaside application.

If you are using the Web Edition of GLASS, using navigation URLs will definitely make it possible to sustain rates in excess of 15 pages/second, since you will be able to avoid saving uninteresing session state in the repository.

For low traffic sites, I think navigation URLs hold promise for making it possible to get the most bang for your resource buck.

The prototype for making the ‘_k’ and ‘_s’ optional parameters is less than a week old and I know that there are some weak spots in the implementation, but I’m sure that they can be resolved:

I don’t like how navigationOnly is used in SushiSession. I’m sure that navigationOnly can be eliminated, but it might be necessary to change the logic for generating Form html, since I added navigationOnly so that Forms would work correctly.

I would like to be able to create Forms where an URL could be used as an alternative to using a #callback:. I know I risk be boiled in oil for this, but I would have liked to include a search field on nearly every page, but that would have required a #callback:.

After spending bits and pieces of 3 months working part time towards the reuse of session state, it is pleasing to see how little code it took to make the ‘_k’ and ‘_s’ parameters optional.