Ken Treis has post up called Deep SIXX with XMLPullParser about how he and the folks at his company use SIXX to transfer objects between GemStone and Pharo.

Ken ported Antony Blakey’s XMLPullParser from VisualWorks to Squeak in a (successful) attempt to reduce the amount of memory consumed while copying large object graphs using SIXX.

If you are developing in Squeak/Pharo and deploying in GemStone, then you should check out Ken’s work.

Photo by jayhem (Creative Commons).

Longtime GemStone customer Intercontinental Exchange has been named 2009 Exchange of the Year by Energy Risk magazine.

(c) Rick Harrison

Earlier today, Ramon Leon posted a very good response to a Pat Maddox question How do you handle constraints on persisted collections?:

I’m trying to make the leap from RDBMS/ORM thinking to pure objects,
and am running into some problems. The first is how to handle
constraints on collections, when the objects inside of that collection
are mutable. Quick example, ensuring that only one user exists with a
certain username. It’s trivial to check that a username is available
before adding the user to a UserRepository. If a user can update his
own username though, it’ll bypass any check on the UserRepository and
invalidate that constraint. A couple options…

1. Have a ChangeUsername command that reserves that username in the
repo, tells the user object to change its username, and then tells the
repo to free the original username
2. When user username: is called, publish a UsernameChanged event that
the repository subscribes to. The repo can choose to reject the
change if it would violate the repo’s constraints.

The first is just a bit ugly and procedural. The second requires the
User object to publish events whenever something happens – which is
easy to forget to do, or you’re left with determining which method
calls warrant the publishing of events. Also you’d have to roll back
the change if the event is rejected.

You could handle that stuff by writing some framework code to manage
the tedious stuff…I don’t love either approach though. How do you
guys tackle this sort of problem?

Pat

And I thought that I would reproduce Ramon’s answer here, because I think it helps folks coming from a relational world to start thinking in terms of persistent objects:

The short answer is you don’t.

The longer answer is you’re still thinking relationally; thinking in sets.
OODB’s and RDBM’s are different paradigms and have vastly different strength
and weaknesses. RDBM’s are much better at guaranteeing data integrity
through things such as constraints and keys rather than collections and
pointers, there’s a theoretical foundation for this, but OODB’s have no such
mathematical rigor. This is a weakness of the OODB. There is no one
provable right way to do it.

On the other hand, RDBM’s have to force everything into a single fixed
schema data structure to accomplish this. The problem is this isn’t the
data structure most programs actually use. RDBM’s don’t speak objects, the
primary method programs actually use to model their domain, and every
attempt to bridge that layer by automatically mapping objects to tables
leads to very serious constraints on the object model. It becomes infected
with compromises to performance or compromises to the flexibility of the
model in order to make it fit. Last but not least you’re stuck with a whole
new layer of code to maintain, the mapping layer, that wouldn’t even be
necessary with an OODB and prior to Rails was often more code than the
domain model itself. If you really enjoy programming with objects,
relational databases will never quite fit right, there’s just an impendence
mismatch that as far as I know no one has ever solved cleanly.

So for a query optimizer, declarative constraints and indexing, you have to
do a whole lot more programming. A fair trade to some for the rigor they
require, but for those who choose the OODB, they’re willing to trade the
rigor for the pragmatism of doing a ton less work and actually getting
things done that just work good enough. I can’t tell you how many RDBM’s
I’ve run across that in production that didn’t have a single foreign key in
place or use constraints at all (often for performance reasons) and chug
along just fine making plenty of money for the gold owner. You’re
experienced with Rails, so you’re aware that most Rails apps don’t use
foreign keys at all. Rails treats the db like a big persistent hash table
preferring to keep the logic in the model. This is a very object way of
doing things. It’s not perfect, but it works well enough in practice.

Look at what Rails has to do to try and fake polymorphic collections,
something that is so utterly trivial in an OODBMs that you don’t even
realize it’s anything special to do. Toss a bunch of object into a
collection and commit; done. The flexibility you gain in modeling things
any way you wish using any data structure you want more than makes up for
the things you lose, i.e. simple declarative constraints and a query
optimizer that lets you just slap on indexes after the fact.

With an OODB you have to think more up front and structure you domain in
such a way that your primary use cases are primarily handled by pointer
navigation. If you’re building a blog, you index the posts by the URL in a
dictionary, you put the comments into a collection in the post. Maybe you
hang the posts off the user who wrote them as well. When it’s all setup,
there’s very little if any querying going on. Now OODB’s do support indexed
collections when you need queries but if you’re writing a behavioral app
rather than a reporting engine, more often than not you don’t want queries
you want tree’s of related objects that actually do things.

If you need to use some special high performance custom written collection
to index your model just the way you want it, an OODB lets you, it’s just
another object after all. If you need collections of objects where every
instance could have wildly different data because you allow the user to add
custom fields, the OODB has no problem with this. It doesn’t care about the
structure of your objects. The price you pay for that flexibility is you’re
on your own, no declarative constraints or query optimizer. On the other
hand, you’re not limited in what you want to do. A relational db just can’t
do user generated fields without converting those fields into rows and
losing its queryability and performance.

Sorry, that turned into a longer post than I’d intended and I’m sure you
already know much of if not all of this, but others who are considering
OODBs might find this useful. It’s a different paradigm and requires a
different style of thinking but pragmatically has a lot of benefits if being
productive is the goal.

Anyway, my approach to your particular issue would be to simply make the
name field immutable after being set the first time. A simple guard clause
in the accessor that threw an exception if someone tried to set the field if
it already had a value and the new value was different would protect any
unintended aliasing bugs. After all, an object should encapsulate its own
business rules and if the field were unique I’d have modeled that by hashing
it in a dictionary by that fields value. It’s not the rigorous guarantee
that a constraint on a set is, but objects aren’t about sets, they’re about
instances and in any real program it’s pragmatic enough to get the job done.

Ramon Leon
http://onsmalltalk.com

[1]

In the month since the last beta update, I’ve been mostly heads down working on Metacello a version control/configuration managements framework. Metacello isn’t ready for public consumption yet, when it is, you will hear about it here. [Update later that day] Metacello is not a Monticello replacement, it is a Monticello enhancement.

Besides Metacello there have been a handful of bugfixes that are worth packaging up. The most important of which gets rid of potential infinite recursion in ExceptionA>>description.

GemStone-dkh.467

No GemTools changes. Added GLASS.230-dkh.231 to update list as an alpha release.

GLASS.230-dkh.231

GLASS bugfixes and additions:

• fix for ExceptionA>>description infinite recursion problem reported by Dario
• non-recursive MCAncestry>>ancestorsDoWhileTrue:
• added methods to be able to cleanUp MethodVersionHistory instances (thanks Gerhard)
• fix problem with recompile:, where method hasn’t been loaded because of other issues
• bugfix 39932 – Url decoding not decoding ‘+’ in url
• fix a problem reading an MCZ reported by Dario … turns out we needed a ‘null’ Error handler:)
• incorporate Squeak protocol additions from Ramon Leon.

Update Process

To update an existing installation to GemStone-dkh.467 and GLASS.230-dkh.231, follow these instructions. If you have recently downloaded an appliance or installed GemStone/S 64 2.3.1, then follow these instructions telling you how to get a recent GemTools client vm.

If you update to GLASS.230-dkh.231 from a version prior to GLASS.230-dkh.231, a Warning dialog will pop up – proceed through the warning.

—–
[1] Photo by dashananda (Creative Commons).

Norbert Hartl just published Easy remote gemstone, an article outlining the steps for setting up a linux host for remote access by the GemTools. Norbert details the steps for either poking holes in your firewall or using an ssh tunnel.

Photo by extratony (Creative Commons).

Andreas Brodbeck published Seaside project is live! a report on his successful use of GLASS and Seaside for a course management tool application.

Andreas shares some of his tips for doing development in Squeak/Pharo and deploying in GemStone/S.

Photo by gorgeoux (Creative Commons).

[1]

Norm Green has announced open enrollment for the following course:

Course: GemStone Performance and Advanced System Management
Dates: June 8 – 10, 2009
Location: Beaverton, Oregon, USA
Cost: $2300 per student.
Registration: send email to training at gemstone dot com

Description:

This three-day course builds on the concepts introduced in GemStone’s
Fundamentals of GemStone/S course. Students gain an understanding of
advanced concepts and issues associated with the performance and
tuning of GemStone’s object server. The instruction format consists of
lectures and guided explorations through actual examples providing
each student with real-world experience.

You Should Take This Course If You

• Want to enhance the performance of GemStone with tuning techniques
• May have some responsibility or interaction with administering a GemStone system
• Seek a deep understanding of the GemStone object server

Your Recommended Prerequisites For This Course Are

• “Fundamentals of GemStone/S” course

This Course Will Teach You About

Concepts

• How a functioning GemStone system behaves
• Fundamental and advanced concepts of GemStone’s architecture
• Sophisticated monitoring and troubleshooting techniques
• How to obtain and apply numerous significant server statistics
• Tuning a GemStone server for performance

—–
[1] Photo by Jeff Blucher (Creative Commons).

[1]

If you find yourself wishing that you could work with SOAP and GLASS, you don’t need to head out to the shed and drag out the old bucket and squeegee. All you need to do is load up the latest preview GLASS version and you’ll be ready to go!

HTTP Server and Client

Before getting started with SOAP, we need to talk a little bit about Hyper. SOAP needs an HTTP client to access services and an HTTP server to provide services. While there are a number of good Smalltalk-based HTTP server implementations floating around out there, there aren’t very many good HTTP Clients!

Fortunately,  Bruce Badger has been continuously updating his Hyper HTTP server and last fall he released a version of Hyper with a very nice HTTP Client implementation. In November, I used the Hyper HTTP Client in a port of Masashi Umezawa’s SoapCore and created a GLASS.231 branch. I needed to create a branch, because the new Hyper was incompatible with the version of Swazoo that I was using for Seaside.

Recently I’ve received a number of requests for SOAP support in GLASS, so I bit the bullet last week and tackled the Seaside/Swazoo/Hyper issue. After a couple of days working through various solutions, I decided to ditch Swazoo and write a simple service class for gluing together Hyper and Seaside. See the section on Using Hyper for details.

AIDA

To be fair, the Swazoo code used in GLASS was several years old (at least). Janko Mivšek has a newer version of Swazoo (Swazoo 2) ported to GLASS, but for SOAP support I need an HTTP Client and the latest Hyper is the ticket. To use Aida with GLASS going forward, you should probably unload the Hyper package, before following Janko’s instructions for running Aida/Web on GLASS.

SOAP

Back in November, I was able to find 3 SOAP servers in the wild that were still up and running (4s4c, Microsoft Soap Interop Server and EasySoap++ – as of this writing 4s4c appears to be MIA). The servers were built based on the ROUND 2 SOAP Interoperability Tests Specification a (proposed) SOAP service interface that can be implemented by SOAP implementations as part of an interoperability testing process. This page is the best resource for information about the Interoperability Tests and this page has the results of the Interoperability Tests as of November 2001. The tests were arranged by the SOAP Builders Yahoo group and this looks like the message that kicked things off.

I was happy to find a set of servers out there that were still alive, since the servers for the original interop tests built into SoapCore were no longer available! I then built a client and server on top of SoapCore (SoapInteropClient and SoapInteropServiceImpl) and a set of TestCases (SoapInteropTestCase) so that I could test the GLASS SoapCore implementation against the 3 external services and an internal service. By default the test cases for the external services are disabled (see SoapInteropTestCase class>>skipInteropTesting: for information about enabling the tests against the external services).

If you find that you need to use SOAP with a project start by looking at the interop examples. Also read the sections GemStone Processes and GCI and SOAP/HTTP Server Development Tips for information on doing SOAP development in GLASS.

SOAP Server Transactions

I have not done any work to add transparent persistence to the SOAP server, but I would imagine that the approach used by FastCGISeaside>>answerResponderRoleCheckingLock: and SeasideHTTPService>>answerToCheckingLock: will work for SOAP. The same rules apply:

• Perform an abort before handling the SOAP request and perform a commit before returning the SOAP response. If no persistent objects were modified while serving the request (a read only request for example), the final commit is no more expensive than an abort.
• One concurrent transaction per gem. Add more gems for handling concurrent requests (presumably Apache or lighttpd can be used to round robin the SOAP requests).
• On a denied lock or transaction conflicts abort and retry the request a short while later.

If anyone is interested in adding transactions to a SOAP server let me know and I’ll whip out a prototype. Right now I assume that the primary interest in SOAP is as a client.

WSDL

Paul DeBruicker attended the GLASS Workshop in Santa Fe earlier this month and recently Paul announced (beta mailing list subscription required) his desire to port the Spray client from Dolphin to GemStone/S. Spray is a Dolphin-based package written by Steve Waring that adds WSDL support to SoapCore.

Nice timing.

Hernan Wilkinson has offered to work on the project along with Paul and I’m sure if anyone else interested in SOAP/WSDL support for GLASS, that your help would be appreciated. When it comes to web servers, I’m sure that the more players involved in interoperability testing the better the final product. Join and send mail to the beta mailing list if you would like to participate.

Using Hyper: SeasideSite replaced by SeasideHTTPService

“Ditching Swazoo” is not quite as bad as it sounds. Anyone using FastCGI will not be affected and the new version of Hyper looks to be an improved version.

If you use Hyper in production, then I would recommend that you move forward cautiously. I haven’t put the new Hyper under stress for this preview release. I intend to do so before making a beta release. Hopefully the next beta release (non-preview) will be the last beta release, so I plan to subject the system to some heavy load for both FastCGI and Hyper. As it currently stands, Hyper passes the unit tests and I’ve done limited manual testing.

If you are using $GEMSTONE/seaside/bin/startSeaside_Hyper, then you need to edit the file and replace the chunk of code that looks like this:

server := SwazooServer singleton.
server initialize.

site := (SeasideSite new)
    name: 'Seaside';
    initialize;
    host: GsSocket getLocalHostName
        ip: (GsSocket getHostAddressByName: GsSocket getLocalHostName)
        port: $1;
    initializeResourceFromUriPattern: '/', 'seaside'.

server addSite: site.

GsFile gciLogServer: 'Hyper Server started on port ', $1 printString.
SwazooServer startSite: 'Seaside'.
"does not return, unless there's an error"

with a chunk of code that looks like this:

server := SeasideHTTPService onPort: $1 acceptInForeground: true multiThreading: true.
GsFile gciLogServer: 'Hyper Server started on port ', $1 printString.
server start.
"does not return, unless there's an error"

If you have been using SeasideSite launchSeaside: 9765 to run Hyper using a GemTools Client, then you should use SeasideHTTPService startServiceOnPort: 9765 instead.

Finally don’t skip the sections GemStone Processes and GCI and SOAP/HTTP Server Development Tips, since they contain valuable information on doing Hyper development in a GemTools Client.

GemStone Processes and GCI

When you do a login from a GemTools Client, a separate Gem process is started on the stone’s machine. The client Smalltalk image and the Gem process communicate via GCI using a relatively simple protocol:

• Commands are initiated by the client (most often sending a message to to an object or executing code in a string). When a GCI command is executed in the Gem a new GsProcess is created and the command is run within the context of that new process.
• While the command is executing in the Gem, the client is blocked waiting for a response (there is a non-blocking mode for GCI which we use for in the GemTools Client, however, the operation is still blocking as far as the end user is concerned, because the GCI API is not thread safe).
• If you hit ALT-. in your Squeak/Pharo image, we catch the client-side interrupt and use the GCI to send a Hard Break to the GsProcess running in the Gem. This results in a GemStone debugger being opened. If you close the debugger the GsProcess is terminated. If you Proceed, the GCI API is used to resume execution of the interrupted GsProcess.
• Unhandled Errors that occur during execution on the Gem are passed over the GCI to the client where we arrange to bring up a GemStone debugger. Note: if you use the [unset|set] UsePopUpOnDebug command on the Admin… menu you can arrange to open a Squeak/Pharo debugger on the client-side Process.
• A command finishes when the GsProcess created to run the command ‘falls off the end’ (like a doIt) or there is an explicit return in the executed code. In either case the result of the command is returned across the wire to the client.
• Execution of code in the Gem blocks until another GCI command is received.

The Gem blocks when a command finishes execution. If you execute a command like:

[[ true ] whileTrue: [ (Delay forSeconds: 10) wait ]] fork.

The command will return immediately and the Gem blocks until you execute another GCI command.

For normal development activities, this doesn’t make too much difference, but if you are trying to debug a web server or a SOAP server, then you need be aware that unless you have explicitly given the Gem control (via a GCI call), your Gem threads will be blocked.

SOAP/HTTP Server Development Tips

A lot of issues can be fleshed out using unit tests, but occasionally you hit an issue that only appears when the system is running as a server. For Seaside applications we’ve arranged for continuations to be used for the remote debugging of application errors.

I have not added continuation-based debugging to the SOAP server (could be done if there is interest), so in order to debug a SOAP server, you’ll need to run your development vm as a server. You need to do the same thing if you want to debug internal server errors for Hyper or FastCGI.

Given that you understand that your Gem threads are blocked:) while your GemTools Client is active and given that you’ve opened a Process Browser you are ready to do server-side development. At the end of this section, I provide instructions for running SOAP/Hyper/FastCGI servers within your development vm.

For the purposes of discussion, I’ll assume that you’ve started a Hyper server and that you’ve navigated to the WAErrorTest page (click on the image for full size view):

Click on the Raise error link and click on the Debug link and the debugger will pop up in the GemTools Client:

The third process involved is the main hyper thread that is listening on port 9765:

So we’ve got three processes of interest: the main accept connection loop for Hyper, a thread forked off by Hyper to service the browser connection, and the a thread forked off by Seaside to service the Seaside request. We’ve got a debugger open on the the third thread and since we signaled an Error, the exception isn’t resumable. If we close the debugger control will be maintained by the client and the Gem execution is blocked.

Once the debugger is closed you can do a couple of things:

• Select each of the other two process and use the terminate menu item to terminate the process. If you go this route, you’ll (obviously) need to restart the Hyper server to serve HTTP requests.
• Select the process that is waiting on a Semaphore and terminate it. If you go this route, you simply need to the use the yield forever menu item on the Process Browser to return contol to the Gem and allow the processes to run freely.
• You should also be able to simply use yield forever without terminating any process – if you look closely you’ll see that the process in the debugger has an #ensure: block that should #signal the semapore that the other process is waiting on. Unfortunately, that doesn’t correctly at the moment. The fact that we’re bringing up a debugger with a hard break, is a special case and I’ll be looking at this bug more closely.

If you hit Alt-. while the server has control, control will be returned to the client, however, you can’t determine which process will be interrupted – the process that is running when the vm detects the interrupt will have the debugger brought up on it.

For all of the server variants, internal server errors are logged in the Object Log – commits are needed in the Soap server to make the log entries persistent. Take a look at the object log after running the Soap unit tests, for examples.

The following three sections provide specific information for debugging server code.

Soap Server

The following expression starts an HTTPServer listening on port 8823 that is ready to serve SOAP requests from the ROUND 2 SOAP Interoperability Tests Specification:

SoapInteropServiceImpl registerAllServices.
(Delay forMilliseconds: 100) wait.

The Soap service is started in the background. so the Delay is necessary to allow the server thread to make enough progress to be ready to service client requests. This setup is aimed at convenience for running tests. For a production server, you should arrange for the HTTPServer to be started running in the foreground (see SoapHTTPService class>>named:onPort:).

If you are trying to debug a Soap server error, here are a couple of useful methods to set breakpoints in:

• SoapHttpConnector>>send:to:with: in the SoapHttpGenericError handler.
• SoapEnvelopeBuilder>>buildSoapEnvelopeFromXmlString: in the Error handler.
• SoapServiceHandler>>dispatch:with: in the Error handler.
• HTTPConnection>>internalErrorHandler.

Hyper Server

The following expression starts an HTTPServer listening for HTTP requests on port 9765:

SeasideHTTPService startServiceOnPort: 9765

If you are trying to debug a Hyper server error, here are a couple of useful methods to set breakpoints in:

• SeasideHTTPService>>internalServerMalfunction:
• HTTPConnection>>internalErrorHandler.

FastCGI Server

For FastCGI you need to edit your Apache (or lighttpd) config to route all requests to a single server. You also need to stop any FastCGI gems that may be running (you can use the Stop Seaside gems command on the Admin… menu). Then run the following expression to start a FastCGI server listening on port 9001 in your development vm:

FSSeasideHandler startUp: 9001.

If you are trying to debug a FastCGI server error, here are a couple of useful methods to set breakpoints in:

• FsSeasideHandler>>internalServerMalfunction:
• FSConnection>>handleError:
• FSConnection>>handleFSError:
  

GemStone-dkh.466

A number of commands were added to the GemTools Launcher.

• [unset|set] UsePopUpOnDebug command added to Admin… menu. If you consistently get an out-of-memory error it may be caused by a recursive error. Setting UsePopUpOnDebug causes a pre-debugger window popup to appear. If the pop-up appears instread of an out-of-memory notifier, select the Debug Squeak stack option and send the Squeak stack as a Bug Report to the beta mailing list for analysis. If you choose Debug GemStone stack the normal GemStone debugger will be opened (although in an out-of-memory situation, it is likely that there is an error opening the GemStone debugger).
• System Browser command added to Tools… menu.
• Mark For Collection command added to Admin… menu.
• Browser Preferences submenu to Admin… menu which allows you to set preferences directly on server instead of inheriting from client (which might require an logout/login sequence).
• Override FFI>>finalize with a bugfix that eliminates that annoying finalize error that occasionally pops up.
• JadeServer class moved to server-side for faster logins. A version of JadServer still exists on the GemTools Client for backwards compatibility with older server versions.

Take a look at the GemTools change log for a complete history of changes since GemStone-dkh.449.

GLASS.230-dkh.229

In addition to the Hyper overhaul and addition of SOAP support:

• finish fixing Bug39828 – fastcgi and uploadfile problems (thanks Dario)
• suspend, terminate, yield 5, and yield forever commands were added to the Process Browser.
• Several packages have been  removed from the GLASS package during the preview process. Normally these orphaned packages remain in your repository unless you explicitly unload them. However, with GLASS.230-dkh.220 I decided that the following 5 packages needed to be explicitly removed (via preDoIt): SwazooSeaside, Swazoo, SqueakSource, MonticelloConfigurations, TinyWiki. If these packages are not explicitly unloaded, the unit tests will not pass.
• JadeServer class moved to server-side for faster logins.
• The package GemStone-Scaffolding was added to GLASS. See James Foster’s Scaffolding for GemStone screencast for more information.
• The #defaultAction for Warning has been changed to bring up a confirmation window which will permit you to continue or bring up a debugger. This means that you’ll start seeing warning messages during Monticello operations that were silent before.

Take a look at the GLASS change log for a complete history of changes since GLASS.230-dkh.216.

Update Process

To update an existing installation to GemStone-dkh.466 and GLASS.230-dkh.229, follow these instructions. While it is possible to use the Monticello Browser for upating, I am now recommending that you use the Update Client and Update GLASS menu items on the Update… menu to avoid issues that may be introduced by UI changes.

When you update to GLASS.230-dkh.229 a Warning dialog will pop up (due to the changes in the #defaultAction for Warning) – proceed through warning.

—–

[1] Photo by sea turtle (Creative Commons).

James Foster has just announced the availability of the Seaside tutorial Learning Web Development with Seaside, that he has been developing for several months. The tutorial is the basis of his popular Weaving a GLASS Web workshop and his SoCal tour.

The tutorial is being made available under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 License.

If you are interested in having a Seaside tutorial/workshop in your area, please contact James at James dot Foster at GemStone dot com.

Photo by Rob Shenk (Creative Commons).

[1]

A couple of weeks ago we published a one-click GemTools 2.4 download. We named the file 2.4 to differentiate it from the 2.3.1 GemTools client download, but we inadvertently included 2.4 gci files in the download – we haven’t released 2.4 yet.

We’ve now made two one-click GemTools preview downloads available on the download page:

• GemTools 2.3.0-preview for all platforms (for use with version 2.3.0 of GemStone/S, like the 1.0beta11 appliance)
• GemTools 2.3.1-preview for all platforms (for use with version 2.3.1 of GemStone/S)

If you are using the GemTools 2.4 download and have experienced problems then you should definitely download and use the one-click file that matches your stone version.

—–
[1] Photo by Nessa Land (Creative Commons).