About a week ago, one of our early beta users pointed out that the single Seaside VM limit (plus a Maintenance VM) in the Web Edition is not practical. Since each http request coming into the Seaside VM acquires the transaction mutex for the duration of the request, a long running http request can effectively block all other requests.
An application developer cannot always predict how long certain requests will take. The developer needs to be able to count on concurrency in the framework, so that a long running request in one session won’t affect the response time for other requests.
We anticipated that users would eventually scale their systems to use multiple Seaside VMs and all along we felt that being able to scale an application by simply adding another VM or 2 to the server is a major advantage, however, we did not anticipate that multiple VMs would be required, which is why we set the 2 vm limit in the first place.
We haven’t made an official announcement (yet), but rest assured that when the Web Edition is released, it will not be limited to 2 VMs. I have to tip my hat to GemStone management. When they were presented with technical arguments against limiting the Web Edition to 2 VMs, they didn’t flinch. They get it that the Web Edition has to have enough punch for people to be successful using the free version.
Multi VM Web Architecture -Round Robin Requests
Most Seaside applications that are running in Squeak (or VW, or Dolphin) arrange for all http session requests to be routed to the same VM, using some flavor of session affinity. Given the GemStone limitation that only one concurrent request can be processed by a GemStone VM, if we were to use session affinity, we would have to allocate one GemStone VM per Seaside session – not acceptable.
Fortunately, with GemStone, Seaside session state (including continuations) can be persisted. With session state stored in the repository, any VM can service any http request and no session affinity is required. The load balancer need only round robin requests to an available VM. The session state for active users stays in the GemStone shared page cache (in shared memory), so each VM can quickly access the data if it is not already resident in it’s own memory space.
It’s relatively straight forward to set up such a load balancing scheme with Apache and FastCGI.
As for sizing considerations, the number of GemStone VMs required for a Seaside application should be based on the number of concurrent requests (not sessions) and the memory allocation for a GemStone VM should be based on the working set requirements for the largest request, not the requirements for the session (i.e., you can choose a session expiry based solely on your application requirements, rather than having to take memory impact of a long session expiry into account).
Given these considerations; the GemStone VMs will be smaller than their Squeak conterparts, but more GemStone VMs will be running in an installation.
I’ll be writing a post on the gory details of setting up Apache and GemStone.
Application Impacts – Concurrency Issues
As I have mentioned in an earlier post, when running with multiple VMs serving Seaside, you will need to worry about transaction conflicts.
A transaction conflict occurs in GemStone when the same object is modifed by two separate VMs at the same time.
You will not have to worry about transaction conflicts caused by modifying your Seaside session state – we preserve the existing Seaside semantics that http requests for the same session are not processed concurrently.
In your current Seaside application, the places where a semaphore is used to protect access to ‘shared’ application data, are sites where transaction conflicts could occur in a multi-vm environment. In your assessment of your application you should also consider the places where you should be using a semaphore, but aren’t.
GemStone has a set of reduced conflict classes (RcCounter, RcIdentityBag, RcKeyValueDictionary, and RcQueue) that can be used to avoid transaction conflicts. We also have object locks that can be used to provide cross-VM access control. Both object locks (on WASession) and RcKeyValueDictionaries are used in the GemStone/Seaside port to avoid transaction conflicts in the Seaside framework data structures.
I’ll be writing a post with more details on transaction conflict avoidance, as well.
All in all, I am very happy to have gotten this kind of feedback early in the process. Our intent is to provide real value with the Web Edition and these changes are an important step in that direction.