You are currently browsing the category archive for the 'Scalability' category.
[Update - 3/19/2009: Read The SSD Anthology: Understanding SSDs and New Drives from OCZ (on AnandTech) if you are thinking of buying an SSD. There's useful information in a discussion of the article over at Hacker News].
Because disk i/o is one of the primary factors affecting the performance of GemStone/S applications, using SSD drives has been an intriguing proposition ever since they popped onto the scene.
About a week ago, Otto Behrens posted a message on the GemStone Mailing list (emphasis mine):
I’m supporting a GS/S 32 bit system at an investment / insurance company. The database is currently 16GB and runs about 40 concurrent sessions.
I recently bought a 80GB Intel® X25-M Mainstream SATA Solid-State
Drive (SSD) and installed it on a Windows XP desktop (Pentium 4, 1.2GB
RAM). I set up a 512MB SPC and copied (all!) the extent files onto the
SSD, leaving tranlogs on the internal IDE. This machine outperforms
the production environment significantly! OK, the production
environment is a Sun with 2 x 1GB CPUs and 8 GB RAM; the GemStone DB
there is set up with an SPC of 3GB.
Anybody tried this with GS/S? I think it will be significant because
the SSD is perfectly suitable for this kind of database. Because of
all the random reads done on these databases, the biggest bottleneck
have always been the seek time on magnetic disks. The SSD reduces seek
time to practically zero! The net result is that we need a much
smaller SPC and can simply use internal SATA disks. Well, let’s see
first, but I can see much cheaper hardware with much better
performance coming…
And yesterday, Hernan Wilkinson wrote in a message (emphasis mine):
Hi Otto, after your mail we decided to give the SSD a try, and what can I
say… thank you very much for sharing your experience with all of us!
We bought a Super Talent Master Drive OX (http://www.supertalent.com/products/ssd_detail.php?type=MasterDrive%20OX#) that
has 150 MB/Sec (max) of Seq. Read, 100 MB/Sec (max) of Seq. Write and 0.1 ms
of access time. As you can see, it is slower than the Intel X25-M and the
results are just outstanding.
Here are some numbers of one of the migration’s step we have to run in a
couple of weeks in one customer. The customer’s extent is about 3.5 Gb. The
process was run in the same machine with the same GemStone configuration (we
just changed the path of the extent and transaction log files to point to
the SSD drive). The hard drive is a Maxtor stm3250310as ( 300 MB/s of data
transfer rate and 11 ms of average seek time). The machine has an Intel Core
2 Duo of 3.0 GHz and 2 Gb of ram (600 MB of SPC), running Windows XP, SP3:
* Looking for all instances of 8 classes:
* With the hard drive: 28 minutes, 47 seconds
* With the SSD drive: just 39 seconds!!
* Migration total time:
* With the hard drive: 35 hours, 32 minutes, 3 seconds
* With the SSD drive: just 58 minutes, 28 seconds!!
So the difference is really big. We are thinking about telling our customers
to switch to this type of disk.
That’s a 30X performance gain achieved by installing SSD drives!
What’s going on?
If the Shared Page Cache (SPC) is too small relative to your working set, performance will be limited by how fast data pages can be read from disk.
If dirty pages are created too fast (high commit rate and/or high data volume), performance will be limited by how fast data pages can be written to disk.
Even if a system runs at an acceptable rate, its equilibrium can be upset by maintenance activities such as:
- Garbage Collection which involves an increased level of page reads, while all object in the repository are scanned, and an increased level of page writes while the dead objects are disposed and pages are reclaimed.
- Data Migration which involves an increased level of page reads, while allInstances are collected, and an increased level of page writes as each instance is migrated to it’s new shape.
The page read problem is generally solved by increasing the size of the SPC (GS/S 32bit SPC is limited to ~4GB). With an SSD drive, read rates are significantly faster so that you can see significant performance gains without changing the size of the SPC. This means you should be able to allocate your excess RAM to OS processes (like more and/or larger VMs).
The page write problem is generally solved by spreading the extent files across multiple disk spindles and adding additional AIO Page Servers. With an SSD drive, write rates are significantly faster so that you don’t need multiple AIO Page Servers (and multiple disk spindles) to keep up with the generation of dirty pages. This means that you don’t have to add external drives to supply the needed spindles.
I haven’t gotten my hands on an SSD drive yet, but with the kind of results that Otto and Hernan have seen it looks like SSD drives should receive serious consideration when you are looking to squeeze more performance from your GemStone/S application.
—–
[1] Photo by Éole Wind (Creative Commons).
[1]
I know that the ‘A‘ in GLASS stands for Apache, but I have to admit that since last spring, I have been using lighttpd almost exclusively for my performance tests.
It all started last fall, when I noticed that every once in a while, I’d get some pretty noticeable flat spots in the performance graphs when running at rates in excess of 100 requests per second. You can easily see the flat spots in the graph at right (running this setup – 4 gems on a 4 core machine with 10 concurrent siege sessions). Peaking at 200 requests/second, but the graph has big swings.
I originally blamed the the flat spots on “contention,” but I didn’t know where the flat spots were coming from.
In May I started tracking down the source of those flat spots. Over a several days of testing I was able to rule out disk I/O, network I/O, and GemStone internals as the source of the contention. All vital signs on all of the systems involved were flatlined – I used separate machines for siege, Apache, and GLASS.
I finally got around to using tcpdump and I was able to see that the last packet to flow between machines before the flat spot was an HTTP request packet heading into the Apache box. The flat spot ended with a packet heading from the Apache box to the Gemstone box. Pretty clear evidence that Apache was the culprit. Without getting into the internals of Apache, I figured that the contention must be an unfortunate interaction between the MPM worker module (which is multi-threaded) and mod_fastcgi.
I asked our IT guys to install lighttpd and you can see the results in the graph at right (32 gems on an 8 core machine with 180 concurrent siege sessions). In this run we’re peaking at 400 requests/second (twice as many cores), but the performance graph is much tighter (standard deviation of 36 for lighttpd versus 60 with Apache) and best of all, no flat spots. Soooo, if you expect to be hitting rates above 100 requests/second, you should be using lighttpd.
Not only is lighttpd performant, but it is pretty easy to setup as well. It turns out that Gemstone/S and FastCGI with lighttpd‘ where he describes how to set up lighttpd for GLASS.
——
[1] Photo: Grandpa’s Shed, Uploaded by a o k on 3 Apr 07, 11.14PM PDT.
I know, I know, at first blush it sounds like a bad idea, but if you let the idea marinate overnight and then sear over red hot mesquite – it ends up being a pretty tasty idea. No that isn’t the sound of vertebrae popping in the background:)
I blame Ryan Simmons. In a comment to my previous post, he innocently asked the question (emphasis mine):
Would it be possible to not comit session state to the stone and use something like Ramons current post on scalling ( http://onsmalltalk.com/programming/smalltalk/seaside/scaling-seaside-more-advanced-load-balancing-and-publishing/ ) to redirect users to the same gem.
Avi Bryant read Ryan’s comment and suggested that running Seaside on GLASS using 1 Session per VM could be a viable technique for avoiding commits.
But, 1 Session per VM?
As I’ve detailed elsewhere, there are good reasons for not using a vm to serve multiple concurrent sessions (without doing a commit per request), but if one were to serve a single session per vm, the good reasons are rendered moot.
With 1 Session per VM, session state does not need to be persisted and voila! no commits due to changes in session state. One should be able to approach the same sort of performance achieved with navigation URLs (i.e., 130 pages/second running on commodity hardware: 2.4Ghz Intel, with 1 dual core cpu, running SUSE Enterprise 10, with 2Gb of ram and 2 disk drives – no raw partitions) while continuing to use rich, stateful Seaside components.
Isn’t 1 Session per VM wasteful?
It depends upon how you look at it. The only thing that might be wasted is memory/swap space. A couple of hundred extra processes on todays hardware is not a big deal unless you are swapping.
If you have 100,000 unique visitors per month and a 10 minute session expiry, you end up needing around 20 sessions. Throw in a fudge factor of 5x and you’re looking at 100 sessions.
100 VMs at 100Mb per VM (tunable) should consume 10Gb of RAM, unless you use mmap (which we do). With mmap, only the memory that is actually used is allocated in RAM. The GemStone/S object manager maps and unmaps chunks of memory on demand so the full 100Mb will only be used if needed and when the memory is no longer needed, it is returned to the system. Without some real benchmarks I can’t tell for sure, but I think it is reasonable to assume that 100 100Mb GemStone vms could run comfortably in 5Gb or less of real memory.
On the flip side, in-vm garbage collection is much cheaper, because only the working set for a single session needs to be swept. When session state for multiple sessions is colocated in the same vm, there is noticeable overhead, so with 1 Session per VM we trade memory for CPU.
Conclusions
We will run some real benchmarks to characterize the tradeoffs between memory, CPU, and commits, but based on back of the envelope calculations it appears that 1 Session per VM is a viable approach for scaling Seaside applications with GemStone/S.
I’m headed to Amsterdam and ESUG on Thursday, but I intend to get busy on this when I get back into Portland in early September, so keep your eyes peeled.
“Nobody expects the Spanish Inquisition!” With that Cardinal Ximinez spun on his well-shined heel and pulled the heavy, copper strapped cell door closed. Laughter echoed in the tunnel as the meager light drifted away with the torch smoke.
I didn’t set out to commit heresy, I suppose noone really does.
Sprawled on a bug infested pallet, locked in a cell deep beneath the Abbey, remnants from my former life draped in ragged tatters from my bruised limbs, I wondered if my simple act of heresy was worth the heavy price I am doomed to pay. I hold scant hope for my redemption, but you gentle reader, you have a chance to learn from my descent into dissent.
Back in April…
…after resolving the last couple of issues with transparent persistence, I turned towards the scalability and performance of Seaside and GLASS. For focus I set a hypothetical goal of 100,000 requests/second – if you’re going to aim for something you might as well aim high.
Since GLASS performs one commit per HTTP request I started out by tuning GemStone/S for maximum commit performance. Before long I realized that I was barking up the wrong tree – if our commercial customers can do say 5,000 commits/second, then I’d need to come up with a 20x performance boost for commit processing to reach 100,000 requests(commits)/second – an admirable goal but not very realistic – we’ve spent 20 years tuning commit performance.
For a 20x performance improvement, I’d just have to pack more bang into each commit or somehow avoid state changes altogether. I’d already explored the ‘more bang per commit’ option way back when I first ported Seaside to GemStone, so I knew that there wasn’t going to be much gained by that route. With regards to avoiding commits, the theory is that avoiding commits for 95% of the pages could result in a 20x gain in performance. The more commits avoided the bigger the multiplier. Now we’re talking!
In May, I started exploring the reuse of session state. By the end of July, I had a proof of concept demonstrating the reuse of 99% of the session state, but I wasn’t real happy with the extent of changes I had to make to Seaside. Nor could I eliminate the decoration chain update used with #call:/#answer: the primary navigation scheme for Seaside. We aren’t going to avoid commits on 95% of the pages without being able to navigate statelessly. I need a navigation scheme that doesn’t require Seaside session state.
Last week (end of July) James Foster and I tossed around some ideas for doing navigation in Seaside without using #call:/#answer: and without using Seaside session state. After playing around with a couple of approaches I settled on the following style of URL, which I’ll call a navigation URL – it’s sorta RESTful, but not too RESTful, it is bookmarkable and it was pretty easy write a component to render pages (the most important – Ha, Ha):
http://example.com/sushi?_n&pg=Home
http://example.com/sushi?_n&pg=Sale
http://example.com/sushi?_n&pg=Catalog
http://example.com/sushi?_n&pg=CartPage
http://example.com/sushi?_n&pg=Item&id=26
I wrote an example Seaside application (Sushi-Example) using the navigation URLs. It is available on GemSource in the GemStone Examples project. As of this writing Sushi-Example-dkh.25 is the version that should be used. The Sushi-Example package can be loaded into either GLASS or Squeak Seaside. There is no styling in the example so don’t expect anything pretty. I encourage you to take a look at the example and give us feedback.
Last Sunday morning I was lying in bed mulling over the problem when it occured to me that if I dropped the ‘_k’ off a navigation URL and figured out how to render the page, I wouldn’t need to save a continuation for each page hit and most importantly I’d avoid a commit for each of those pages as well! After a little more thought, I figured it would be possible to avoid saving sesssions making it possible to drop the ‘_s’, too. By Monday afternoon, I had a prototype running.
I spent Tuesday doing some benchmarks to see if there were noticeable improvements in performance. I knew that by avoiding commits I would greatly improve the scalability of Seaside, but I was also hoping to see some performance gains, too.
I have to admit that advocating the use of navigation URLs and proposing the ‘_k’ and ‘_s’ be optional URL parameters makes me feel like a Seaside heretic…
(JARRING CHORD. The door flies open. In come three evil types in red robes.)
A Case to make ‘_k’ and ‘_s’ optional using Navigation URLs
… but here goes, the rack be damned.
I have glossed over a couple of details in the following section, so you’ll want to look at the code for the real skinny:)
In a normal Seaside application URLs are generated for links that look something like this: http://example.com/sushi?_s=68pqfS&_k=SW7A&_n. To decode this URL Seaside uses the ‘_s=68pqfS‘ to lookup a session, the ‘_k=SW7A‘ to look up a continuation in the session and the rootComponent associated with the continuation is rendered. The ‘_n’ indicates to Seaside that no redirect is needed for the rendering pass.
Consider a navigation URL that looks like this: http://example.com/sushi?pg=Item&id=26. To decode the navigation URL, Seaside creates a default session (no ‘_s’ present) and renders the rootComponent associated with a default rendering continuation (no ‘_k’ present). The rootComponent uses the ‘pg=Item&id=26′ to lookup and render the appropriate item.
When a navigation URL is used to reach a page, Seaside creates and saves a new instance of the session class. If a customer is window shopping (i.e., navigating around the site without logging in or adding an item to a cart), the session state is uninteresting and doesn’t really need to be saved. If a request URL has no ‘_s’ and no state is changed while processing the URL, the ‘_s’ could be dropped from generated navigation URLs with no loss of information. There’d be a definite advantage for web sites with lots of window shoppers as only the interesting sessions would be saved.
If a customer logs in or adds an item to a cart, then the session becomes interesting and it makes sense to propogate the ‘_s’ to all generated navigation URLs. Such an URL looks like this: http://example.com/sushi?pg=Item&id=26&_s=68pqfS&_n. To decode this URL, I would expect Seaside to use the ‘_s=68pqfS‘ to lookup a session and render the ‘pg=Item&id=26′ item. The ‘pg=Item&id=26′ is retained in the URL to make it possible to lookup and render the target page. Essentially the ‘pg=Item&id=26′ replaces the ‘_k’, which ends up serving two useful purposes:
- In a window shopping scenario, the customer will continue to navigate around the site. The interesting session state is encoded in the ‘_s’ and without a ‘_k’ there is no need to create and save session state for each page rendered.
- such a navigation URL is natively bookmarkable. If the session expires before the bookmarked URL is used, it still contains enough information to allow a valid navigation to the bookmarked page.
With navigation URLs, a ‘_k’ is needed only when a #callback: or #call: is associated with a rendered anchor or form.
Old Lady : I don’t understand what I’m accused of.
Ximinez : Ha! Then we shall MAKE you understand! Biggles! Fetch… …THE CUSHIONS!
The Benchmark
In the Sushi Example, you can use either the standard WASession or the SushiSession. When you use WASession, you’ll get ‘_s’ and ‘_k’ parameters generated for every link (along with the associated session state), while using the SushiSession, the ‘_s’ and ‘_k’ parameters are only included in the URL when needed. Since I wanted to measure the effect of the accumulation of session state (or lack thereof), I decided to run the tests for 20 minutes (double the default sessionExpory). With a 20 minute test we’ll generate a full complement of session state and presumably settle into a steady state as aged sessions expire.
For all of the tests (except Run5), I used the following siege invocation:
siege -b -d 0 -c 10 -t 20M http://example.com/seaside/examples/sushi
The arguments tell siege to hit the URL as hard as possible simulating 10 concurrent users for 20 minutes. For more on siege, see my post Scaling Seaside with GemStone/S.
For Run5, I had to increase the -c argument in order consume both CPUs on the box. Run5 was made with 300 concurrent sessions.
The Benchmark Results
The following table summarizes the results:
| Run | Req/Sec | Core | Gem | VM | Machine | Session Class |
| 1 | 11 | 1 | 1 | S | laptop | WASession |
| 2 | 14 | 1 | 3 | G | Foos | WASession |
| 3 | 67 | 1 | 1 | S | laptop | SushiSession |
| 4 | 129 | 1 | 3 | G | Foos | SushiSession |
| 5 | 335 | 2 | 3 | G | Foos | SushiSession |
Run1 (Squeak) and Run2 (GemStone) are baseline runs and if you look at the results from my benchmarks last fall, you’ll see that the figures are comparable (previous Squeak results and previous GemStone results).
Things get interesting when you look at Run3 (Squeak). Not saving session state results in a 6x improvement in performance. Since both runs (Run1 and Run3) were made against the same Sushi-Example the difference in performance can be attributed to the reduced load on the memory management system. While I didn’t measure the size of the VM, I am sure that it is a lot smaller in Run3.
For GemStone Run4 the improvement is even more dramatic – a 9x improvement. For GemStone/S, the improvement is due to the fact that without saving session state, no commit processing overhead is incurred. I should note that on the machine Foos, no effort was made to use raw tranlogs (see Scaling Seaside with GemStone/S and GemStone 101: Transactions for info about raw tranlogs and performance) so the improvement is more dramatic than you’d see on a system that was tuned for commit performance.
For Run5 (GemStone) I added an adidtional CPU allowing the stone and 3 gems to utilize both CPUs available on the machine. This time the improvement was 2.5x. You’d expect a 2x, so the extra margin of performance is probably due to the fact that the there was very little context switching going on with 2 CPUs.
Conclusions
The quest for 100,000 pages/second doesn’t look quite as quixotic now as it looked a week ago. I think that using navigation URLs with Seaside is a valid technique especially if you are interested in a highly scalable Seaside application.
If you are using the Web Edition of GLASS, using navigation URLs will definitely make it possible to sustain rates in excess of 15 pages/second, since you will be able to avoid saving uninteresing session state in the repository.
For low traffic sites, I think navigation URLs hold promise for making it possible to get the most bang for your resource buck.
The prototype for making the ‘_k’ and ‘_s’ optional parameters is less than a week old and I know that there are some weak spots in the implementation, but I’m sure that they can be resolved:
I don’t like how navigationOnly is used in SushiSession. I’m sure that navigationOnly can be eliminated, but it might be necessary to change the logic for generating Form html, since I added navigationOnly so that Forms would work correctly.
I would like to be able to create Forms where an URL could be used as an alternative to using a #callback:. I know I risk be boiled in oil for this, but I would have liked to include a search field on nearly every page, but that would have required a #callback:.
After spending bits and pieces of 3 months working part time towards the reuse of session state, it is pleasing to see how little code it took to make the ‘_k’ and ‘_s’ parameters optional.
Before we point jcrawler at a Seaside application I would like to talk about what you should expect.
To start with, jcrawler is like a bull in a china shop. The algorithm jcrawler uses is not very deterministic nor is it discriminating, but it is thorough, relentless and highly parallel. Given an an initial set of URLs, jcrawler traverses each page and adds the links it finds to its list. Every so often, jcrawler creates a new thread to process another URL from the list. We can depend upon jcrawler to rattle every piece of china in an application and it will rattle more than one piece at a time, so we’d better be ready to deal with wreckage.
Jcrawler will help make your application bullet proof, but at potentially 15 errors/second spread across several vms, there can be a lot of wreckage to sift through.
I added an object log to GLASS a couple of weeks ago and over the last couple of days, I’ve added a Seaside application for viewing and manipulating the object log. Take look at a sample log 
(my blog is wide-image challenged). It’s not the purtiest page this side of the Mississippi (I am web-design challenged:), but it does the job.
In the object log, the entries labeled ‘– continuation –, 
represent object log entries that can be debugged via the ‘Debug’ button in the GemStone/S Transcript Window. If you take a peek at the pid column, you will notice that the log entries were generated by two different gems. There are three gems serving HTTP requests in the appliance.
The upshot is that after letting jcrawler hammer on your application, not only do you get an overview of the problems uncovered during the run, but you can open a debugger and investigate the issues that resulted in walkbacks.
I generated the sample object log by manually playing with the randomError application (http://localhost/seaside/examples/GemStone/randomError in the appliance) found in the GemStone Examples project. This little gem generates a simple log entry (’random error’) or walkback (’– continuation –’) 12% of the time. You can also generate different kinds of errors by poking the links in the Error tab of the alltests application (http://localhost/seaside/tests/alltests in the appliance).
If you want to play with the object log, load up the latest version of the GLASS package (GLASS-dkh.103 in the GLASS project – it will also load the GemStone Examples). Poke around in the randErrror application until you get an error then head on over to the object log (http://localhost/seaside/tools/objectLog in the appliance). You can also try the remote debugger from your development image.
Next up we’ll talk a little bit about configuring jcrawler.
Avi Bryant has an article up on his blog where he does a very good job describing GemStone’s architecture. It’s definitely worth a read.
jcrawler is a good tool for load testing Seaside applications – in theory. Unfortunately, it takes a little bit of work to modify jcrawler to get it to the point where it can be used to load test Seaside applications. So here’s my story…
The Story
In the last week I’ve started putting together some examples of persistence for GLASS (I’ll write a post about them when I’m happy with the examples). As I have mentioned before, you need to use different techniques to manage concurrency in GemStone/S, because you will be using multiple vms to serve web pages and Semaphores can’t be used. The examples illustrate several of the techniques that you can use to avoid transaction conflicts. As part of the exercise, I needed to find a way to test for transaction conflicts.
In order to create a transaction conflict, you need to have two web requests hit your web server at exactly the same time. I’ve used siege in the past for load testing, but siege uses constant URLs, not too useful for banging arbitrary URLs buried in the depths of your Seaside application.
To effectively beat on a Seaside application (especially if you want to expose concurrency bugs) you need a load tester that will crawl through your site, pick up the dynamically generated URLs and feed them back into the mix.
I knew that WAPT had been used by several folks for Seaside Load Tests, but I didn’t see site crawling mentioned in the feature list for WAPT. Beside that I’m doing my work on Linux boxes, so a Windows-only tool would not be convenient.
Without trying too hard, I found a site that listed a ton of Web Test Tools and up near the top of the of the Load and Performance Test Tools section there was a listing for jcrawler:
An open-source stress-testing tool for web apps; includes crawling/exploratory features. User can give JCrawler a set of starting URLs and it will begin crawling from that point onwards, going through any URLs it can find on its way and generating load on the web application. Load parameters (hits/sec) are configurable via central XML file; fires up as many threads as needed to keep load constant; includes self-testing unit tests. Handles http redirects and cookies; platform independent.
Just the ticket, huh? Well, if it was that easy, I wouldn’t be writing a blog post would I? Haha!
The Work
I grabbed the download from SourceForge and proceeded to build jcrawler.
You need ant, too. But that’s easily fixed.
The build completed and I was ready to slam my Seaside apps and its only been a couple of minutes! But run.sh failed:
Exception in thread “main” java.lang.UnsupportedClassVersionError: com/jcrawler/Main (Unsupported major.minor version 49.0)
It turns out that you must use JDK 5.0. Another download and some monkey business with my environment variables:
export JAVA_HOME=/home/dhenrich/jdk1.5.0_14
export PATH=$JAVA_HOME/bin:$PATH
and we’re off to the races. I launched jcrawler against:
http://172.16.172.138/seaside/examples/persistence/rcTally
a variant on WACounter running on my copy of the appliance.
The Problem
Things appeared to running okay. jcrawler was spinning away dumping log entries like the following to stdout (sorry about the line wrapping):
2568 [THREAD#93 CREATED 10:40:01::602] INFO com.jcrawler.UrlFetcher – Fetching URL http://172.16.172.138/seaside/examples/persistence/serial?
_s=lKtxmydVdpOJAjpt&_k=ggcZGlQC&1
However, as I interactively poked at rcTally, I noticed that jcrawler wasn’t hitting the + + or - - links, because the shared value was not getting updated.
After an excruciating amount of debugging, I noticed that the URLs extracted from the web page contained the sequence ‘&‘ instead of ‘&‘…geez it has been just about as hard to get WordPress to display the dang ‘&‘ string in my post (can’t use rich editor) as it was to find the problem in jcrawler (sorry about line wrapping):
http://172.16.172.138/seaside/examples/persistence/serial?
_s=lKtxmydVdpOJAjpt&_k=ggcZGlQC&1
The Fix
- Download an HTML Parser from SourceForge.
- Copy the jars from the HTML Parser into jcrawler:
cd /home/dhenrich/htmlparser1_5/lib
cp *.jar /home/dhenrich/jcrawler/lib - Edit the jcrawler source and insert the following line after line 120 in com/jcrawler/UrlFetcher.java (in the jcrawler src directory) to convert the encoded HTML:
content = org.htmlparser.util.Translate.decode(content);
- add htmlparser.jar to the list of jars in run.sh (in the jcrawler misc directory).
- Rebuild jcrawler and you are off to the races!
The Payoff
At the end of the day, you’ve got yourself a version of jcrawler, that can be used to randomly poke around in the nooks and crannies of your Seaside application and give it a pretty thorough workout.
As I work on the GemStone examples, I’ll learn more about jcrawler’s quirks and features, but for now it does pretty much what I need.
If there’s another load tester out there that can crawl through a Seaside website, I’d appreciate hearing about it.
Herb Sutter has just published another article in his series on Effective Concurrency: Use Lock Hierarchies to Avoid Deadlock. Coming on the heels of my post on Transaction Conflicts where I talk about object locks, it is very timely.
Many of you have heard that there are GemStone/S applications running in production with thousands of vms and that other applications are doing thousands of commits per second. I bet you have been wondering what kind of performance you could get running Seaside on GemStone/S – I certainly was:)
Seriously, I spent most of the summer working on tools and with the exception of a few small tests, I didn’t take the time to focus on performance. When I came back from ESUG, I decided to run some scaling tests.
[For those of you who want to see the results and move on with your surfing, here are a couple of links: Goals, Results, Conclusions].
Background
From the tests that I had run over the summer it appeared that Apache was a limiting factor when trying to run at rates above 30 requests per second. I’d also seen some anomalies in the i/o on Linux, were we’d get flat spots in our performance graphs that appeared to be related to file system buffer flushing. If you have read the post on transactions, then you know that we write tranlog records on every commit (page request), so disk i/o can be a limiting factor.
With the help of our IS guy, it turned out that to get the best performance from Apache, we needed to use the MPM worker module. With the MPM worker module turned on, Apache performance fell way off the radar.
The issue with the i/o anomalies that we observed in Linux has not been as easy to resolve. I spent some time tuning GemStone/S to make sure that GemStone/S wasn’t the source of the anomaly. Finally our IS guy was able to reproduce the anomaly and he ran into a few other folks on the net that have observed similar anomalies.
At this writing we haven’t found a solution to the anomaly, but we are pretty optimistic that it is resolvable. We’ve seen different versions of Linux running on similar hardware that doesn’t show the anomaly, so it is either a function of the kernel version or the settings of some of the kernel parameters. As soon as we figure it out we’ll let you know.
For the purposes of these performance tests, I was able to work around the i/o anomaly by putting extents on raw partitions. In nearly all of the tests, the Shared Page Cache (SPC) is sized large enough to hold the entire working set for the test. Consequently there was very little read activity and the system was able to write dirty pages from the SPC fast enough so that random i/o to the raw extent partitions didn’t affect test results.
Goals
I had three goals in mind when I ran this set of tests.
- demonstrate anticipated performance of the GLASS appliance.
- demonstrate production performance for the Web Edition.
- demonstrate performance potential beyond the Web Edition.
For the GLASS applicance tests I wanted to illustrate what you could expect if you installed the VMWare image on a machine without paying particular attention to disk configurations. To simulate the GLASS applicance, I simply ran the tests using file-based tranlogs.
For the production-scale performance of the Web Edition, I wanted to illustrate what you could expect if you paid attention to the disk configuration (i.e., created some raw partitions and had a box with a minimum of 4 disk spindles).
I also wanted to illustrate what kind of performance improvements you could see if you were to add additional SPC or increase the number CPUs available.
Finally, based on a comment where Ramon Leon suggested I include some ’speed comparisons between GemStone and Squeak’, I’ve included runs against a Squeak vm.
Test Strategy
I decided to base the performance tests on the Seaside Counter example. Since the Counter has dead-simple render logic and no significant application state, it is the perfect application for measuring baseline Seaside performance. For GemStone that means that when running tests against the Counter, we’ll be getting performance numbers that include the overhead of persisting and sharing session state (about 250 objects or 50k bytes per request).
Over the course of the summer I ran across siege, “an http regression testing and benchmarking utility” that is very easy to use. It can smack the heck out of a Web Application without putting too much of a load on the system running the test. It also provides some very basic stats about how your app withstood the barrage. Response Time, Transaction rate, and Concurrency being the most interesting.
Siege basically arranges to fire a number of concurrent requests at a given URL. In benchmark mode, as soon as a response is recieved another request is launched in its place. In this mode you can force an application to its knees – very nice for finding bottlenecks. In internet mode, siege waits a random amount of time before firing off the follow-on request, making for a simulation of what the end users of your application may experience.
I ran my tests using the URL ‘http://penny:8000/seaside/examples/counter’, which, as many of you Seasiders out there will recognize, means that a new Seaside session is created on each hit. For benchmarking purposes, that’s just fine – a little extra load never hurts. In the future, I plan to run some scaling tests that measures intra-session performance.
For quite a while now, I have felt that it was important for folks to plan on running multiple vms when they go into production using GLASS. For the best overall response times, one should plan on running at least one vm per concurrent request, which in practice should be 10 or more. For the benchmark tests, the Counter application is cpu bound, so when siege goes about slamming the web server into the ground the cpus are pegged and cpu contention between the processes becomes a factor. It turns out that with 10 vms running on a single cpu, there is a whole lot of contention going on. So after playing around a bit, I settled on using 5 vms for the benchmark tests while running with 10 concurrent requests. This combo gave good performance numbers in the single core tests while in the 4 core test when all 4 cpus were redlined we would minimize the amount of contention. I also ran a couple of internet tests using a single CPU and 20 vms to confirm that in the wild you could afford to run with more than 5 vms without suffering a performance hit. Finally I ran a benchmark test with 100 concurrent requests to see how the system behaved when it was being slashdotted.
Test Setup
For the hardware I used 3 machines foos, toronto, and penny.
Penny is a 2.6Ghz AMD Opteron, with 2 dual core cpus, running SUSE Enterprise 10, with 8Gb of ram. Penny was used to host Apache and Siege. We had a dedicated 1Gbs ethernet connection between penny and toronto.
Foos is a 2.4Ghz Intel, with 1 dual core cpu, running SUSE Enterprise 10, with 2Gb of ram and 2 disk drives (no raw partitions). Foos was used to simulate the GLASS appliance performance.
Toronto is a 2.2Ghz AMD Opteron with 2 dual core cpus, running SUSE Enterprise 10, with 8Gb of ram and 5 disk drives (raw partitions installed on 3 of the partitions). Toronto was used to simulate a typical production machine.
Siege was pointed at the Apache instance listening on port 8000. In addition to the mpm_worker_module, mod_proxy_balancer was used to round-robin requests to the various vms. GemStone was running with version Seaside2.8g1-dkh.490 of Seaside.
For the Squeak tests I used the latest development image from Damien Cassou (sq3.9-7067dev07.10.1), the 3.9 vm and loaded Seaside2.8a1-lr.492. I pointed siege directly at the Squeak image (both running on Toronto).
I did not enforce exclusive use on any of the machines (penny and toronto are shared by other folks in the company) during the tests. But between running most of the tests multiple times and keeping an eye out for anomalous events the numbers are good enough for government work.
Summary of Results
In all, I ran 15 different tests in 6 different categories (Squeak baseline, Squeak internet, Squeak benchmark, GemStone Web Edition, GemStone internet, and GemStone benchmark).
The following table is sorted by Req/Sec. Click on a Run number to jump to the category and a description of the test.
| Run | Req/Sec | Core | Gem | VM | std | Siege | Machine | Notes |
| 1 | 10 | 1 | 1 | S | - | -b -c 10 | Toronto | 1.0 ART |
| 2 | 15 | 1 | 5 | G | 5 | -b -c 10 | Foos | file-based, 1G SPC |
| 3 | 16 | 1 | 1 | S | - | -i | Toronto | 0.3 ART |
| 4 | 25 | 2 | 5 | G | 3 | -b -c 10 | Foos | file-based, 1G SPC |
| 5 | 28 | 1 | 20 | G | 5 | -i | Toronto | raw, 1G SPC |
| 6 | 28 | 2 | 20 | G | 5 | -i | Toronto | raw, 5G SPC |
| 7 | 29 | 1 | 1 | G | 6 | -i | Toronto | 0.02 ART, raw,1G SPC |
| 8 | 32 | 1 | 1 | S | - | -b -c 1 | Toronto | 0.03 ART |
| 9 | 50 | 1 | 5 | G | 10 | -b -c 10 | Toronto | raw, 1G SPC |
| 10 | 75 | 1 | 5 | G | 13 | -b -c 10 | Toronto | 0.1 ART, raw, 5G SPC |
| 11 | 87 | 1 | 5 | G | 6 | -b -c 100 | Toronto | 1.3 ART, raw, 5G SPC |
| 12 | 91 | 1 | 1 | G | 6 | -b -c 10 | Toronto | 0.1 ART, raw, 1G SPC |
| 13 | 140 | 2 | 5 | G | 20 | -b -c 10 | Toronto | raw, 5G SPC |
| 14 | 185 | 3 | 5 | G | 37 | -b -c 10 | Toronto | raw, 5G SPC |
| 15 | 230 | 4 | 5 | G | 40 | -b -c 10 | Toronto | raw, 5G SPC |
ART in the Notes column is shorthand for Average Response Time, a stat from siege.
Squeak Results
Run1, Run3, and Run8 are scaling tests against the Squeak image. Run7 and Run12 are comparable tests run against a GemStone vm. Note that there is only 1 GemStone vm being used in these tests.
| Run | Req/Sec | Core | Gem | VM | std | Siege | Machine | Notes |
| 1 | 10 | 1 | 1 | S | - | -b -c 10 | Toronto | 1.0 ART |
| 3 | 16 | 1 | 1 | S | - | -i | Toronto | 0.3 ART |
| 7 | 29 | 1 | 1 | G | 6 | -i | Toronto | 0.02 ART, 1G SPC |
| 8 | 32 | 1 | 1 | S | - | -b -c 1 | Toronto | 0.03 ART |
| 12 | 91 | 1 | 1 | G | 6 | -b -c 10 | Toronto | 0.1 ART, raw,1G SPC |
Baseline
Run8 showed the best results for Squeak with a rate of 32 request/second when hit with a siege from a single user.
Internet tests
For the internet test (Run3), the Squeak vm hit 16 requests/second (0.34 seconds average response time and an average of 6 concurrent requests). In a comparable test (Run7), the GemStone vm hit 29 requests/second (0.02 seconds average response time and an average of 0.6 concurrent requests).
Benchmark tests
When 10 concurrent users slammed the Squeak vm (Run1), performance dropped to 10 requests/second. Siege doesn’t collect stats on the standard deviation, but I observed a wide range of response times around the 1 second average reponse time. In the comparable GemStone test (Run12), the GemStone vm hit 91 requests/second, a standard deviation of 6 and an average response time of 0.1 seconds.
Under load it appears that GemStone is about 10 times faster processing Seaside requests than Squeak (Run1 comparied to Run12). While the GemStone vm is certainly faster than the Squeak vm, I don’t think that the GemStone vm is that much faster. I haven’t tried to analyze what might be going on, but my best guess is that under load, the garbage collector is siphoning cpu cycles away from the processing of requests.
GemStone Results
Web Edition tests
Run2, Run4, and Run9 are intended to illustrate the kind of performance you might expect when running a version of the Web Edition (i.e., 1 core, 1G SPC, and a 4G extent).
| Run | Req/Sec | Core | Gem | VM | std | Siege | Machine | Notes |
| 2 | 15 | 1 | 5 | G | 5 | -b -c 10 | Foos | file-based, 1G SPC |
| 4 | 25 | 2 | 5 | G | 3 | -b -c 10 | Foos | file-based , 1G SPC |
| 9 | 50 | 1 | 5 | G | 10 | -b -c 10 | Toronto | raw, 1G SPC |
Run2 and Run4 used file-based tranlogs and extents and Run9 used raw tranlogs. You can see that Run9 is over 3 times faster than Run2. Using raw I/O for tranlogs makes a big difference.
Even with 2 cores, Run4 is still slower than Run9. Confirming that with file-based tranlogs, the test is i/o bound.
A sustained rate of 15 requests/second (24×7) is about the top rate that we’d recommend when using the Web Edition. In Run2 the disk-based garbage collector kept pace with the expiration of session state consuming roughly 1/5 of the 4G repository before it was garbage collected. During Run10 (at a rate of 75 requests/second) a 4G extent was consumed in 15 minutes!
Internet tests
Run5 and Run6 illustrate what you might expect for production performance with real world loads.
| Run | Req/Sec | Core | Gem | VM | std | Siege | Machine | Notes |
| 5 | 28 | 1 | 20 | G | 5 | -i | Toronto | raw, 1G SPC |
| 6 | 28 | 2 | 20 | G | 5 | -i | Toronto | raw, 5G SPC |
The difference between Run5 and Run6 is that I used 2 cores and a 5G SPC in Run6. It is clear that neither a larger SPC or more cores are needed to sustain this rate.
These tests were run using 20 vms. In GemStone/S a vm handles a single http request at a time (each http request coming into the Seaside vm acquires the transaction mutex for the duration of a request), so in order to get concurrent handling of requests you need to have multiple vms running. A quick look at Run7, which runs at the same rate with a single vm, shows that you don’t sacrifice performance by spreading the load across 20 vms, while gaining the ability to handle up to 20 requests concurrently.
Benchmark tests
Run10, Run11, Run13, Run14, and Run15 are intended to illustrate how GemStone/S stands up to siege in benchmark mode and to illustrate the scaling performance as you add cores into the mix.
| Run | Req/Sec | Core | Gem | VM | std | Siege | Machine | Notes |
| 10 | 75 | 1 | 5 | G | 13 | -b -c 10 | Toronto | 0.1 ART, raw, 5G SPC |
| 11 | 87 | 1 | 5 | G | 6 | -b -c 100 | Toronto | 1.3 ART, raw, 5G SPC |
| 13 | 140 | 2 | 5 | G | 20 | -b -c 10 | Toronto | raw, 5G SPC |
| 14 | 185 | 3 | 5 | G | 37 | -b -c 10 | Toronto | raw, 5G SPC |
| 15 | 230 | 4 | 5 | G | 40 | -b -c 10 | Toronto | raw, 5G SPC |
Run10, Run13, Run14, and Run15 show a nearly linear progression in performace as more cores are added.
With Run11 I cranked siege up to 100 concurrent requests to see what would happen. The fact that Run11 averaged a little more than Run10 even thought they are running on the same configuration, means to me that the cpu wasn’t running flat out in Run10. With 100 concurrent requests, though, the cpu was truly hammered. If you look at the Average Response Time for the two runs, you’ll see that with 10 times more concurrent requests, it took 10 times longer to respond to a request, which makes sense since the extra concurrent requests ended up being queued up behind the 5 vms.
At rates approaching 200 requests/second, I started seeing indications that the data structure used to store session state (RcKeyValueDictionary) was reaching the limit of its effectiveness. As we move forward I will be looking at data structures that are better suited to these rates.
Conclusions
You can expect very reasonable performance numbers with the Web Edition – pretty good value for your money:). Along with transparent persistence, the Web Edition will support rates up to 50 requests/second across a large number of vms. You will have to keep an eye on the size of your repository, if your sustained rate starts averaging near 15 requests/second. But hey, you can serve an awful lot of donuts at these rates.
If you start getting more traffic, it looks like GemStone/S can be scaled to some pretty respectable rates without having to change your application.
Herb Sutter has just published his second article on Effective Concurrency in DDJ entitled “How Much Scalability Do You Have or Need?”. Check it out
