Metacello: Package Management for Monticello

[1]

[Updated: 6/5/2010] For the latest documentation and information about Metacello, see the Metacello project pages.

Metacello is a package management system for Monticello. A package management system is…

…a collection of tools to automate the process of installing, upgrading, configuring, and removing software packages from a computer. Distributions of Linux and other Unix-like systems typically consist of hundreds or even thousands of distinct software packages; in the former case a package management system is nice, in the latter case it is essential.

Packages are distributions of software and metadata such as the software’s full name, description of its purpose, version number, vendor, checksum, and a list of dependencies necessary for the software to run properly. Upon installation, metadata is stored in a local package database.

A package management system provides a consistent method of installing software…

Last April I finally became fed up with the lack of a decent package management system for Monticello and decided to write my own from scratch. I wanted a package management system for Monticello that was consistent with the important features of Monticello:

• Declarative modeling. A Metacello project has named versions consisting of lists of explicit Monticello package versions. Dependencies are explicitly expressed in terms of named versions of required projects. A required project is a reference to another Metacello project.
• Distributed repositories. Metacello project metadata is represented as instance methods in a class therefore the Metacello project metadata is stored in a Monticello package. As a result, it is easy for distributed groups of developers to collaborate on ad hoc projects.
• Optimistic development. With Monticello-based packages, concurrent updates to the project metadata can be easily managed. Parallel versions of the metadata can be merged just like parallel versions of the code base itself.

Additionally, the following three points are important considerations for Metacello:

• Cross platform operation. Metacello must run on all platforms that support Monticello: currently Squeak, Pharo and GLASS.
• Conditional Monticello package loading. For projects that are expected to run on multiple platforms, it is essential that platform-specific Monticello packages can be conditionally loaded. At the moment, conditional loading is specified based upon the following attributes:
  • #common. Code that is common across all platforms.
  • #squeakCommon. Code that is common to Squeak and Pharo.
  • #squeak. Code that is specific to Squeak.
  • #pharo. Code that is specific to Pharo.
  • #gemstone. Code that is specific to GemStone.

  It should be possible to inject project-specific attributes, so code that is dependent upon attributes not covered by the standard list can be conditionally loaded. For example, in GLASS, different Monticello package versions are loaded based on which version of GemStone/S is running (i.e., version 2.0 versus 3.0).

• Compatible with MC2. It must be possible to manage Metacello projects that are based on alternate Distributed Source Code Management systems like Monticello2.
• MIT license.

[Update: 10/14/2009] Note that with the release of version 1.0-beta.3 of Metacello, some of the following sections are no longer valid . Please read this post for up-to-date information. The outright invalid entries have been struck out.

In the following sections I hope to give a taste of what Metacello can do along with instructions on loading Metacello into Squeak or Pharo, so you can run through the Metacello tutorial:

• The VersionSpec
• Metacello Tools
• GLASS Configuration
• Bootstrapping Metacello
• Tutorial
• Current Status

The VersionSpec

[Update: 10/14/2009] the following section is not completely accurate for the newer versions of Metacello (1.0-beta versions). Please see this discussion for up-to-date information.

Click on the image below to get a full depiction of a version spec.

The metadata for a Metacello project version is kept in a versionSpec which has the following atributes:

• blessing. A version can be tagged with a blessing like #alpha, #beta, #release, #development (or any other tag that is deemed useful). The blessing is used as a version filter. For example the latest version of a Metacello project is currently defined as the latest version whose blessing is not #development.
• description. Useful information about the version.
• packages. A list of Monticello package versions (or project references via required projects) that make up the project.
• repositories. A list of Monticello repositories from which the packages can be loaded.
• groups. An alias for a collection of packages. See Bootstrapping Metacello for some examples of using groups.
• doits. A collection of blocks associated with a Monticello package that are evaluated before and/or after a package is loaded.
• project package. A definition of the Monticello package name and Monticello repository from which the latest project metadata is be loaded.
• required projects. A list of projects that the Metacello project depends upon. A required project spec includes such information as the version of the project, the name of the Metacello project class, the package and repository from which the project metadata is loaded (note the similarity to the project package specification). Heres’ an example of a required project specification:

Any one of the versionSpec attributes may be conditionally modified. Here’s an example package specification that adds  a new Monticello package version for the package ‘Example-Core’ when the #testPlatform attribute is present:

In the #common version spec, the package version ‘Example-Core-anon.14’ is associated with the ‘Example-Core’ package . When the #testPlatform attribute is present, ‘Example-Core.testplatform-anon.15’ overrides ‘Example-Core-anon.14’ in the version spec.

Metacello Tools

[Update: 10/14/2009] Note some of the entries in this section are invalid for later versions of Metacello. Please read this new section for more up-to-date information.

I have developed a minimal set of menu items for working with Metacello.

I started out thinking that I would completely avoid the tools issues and instead rely on workspace scripts, the Class Browser and Monticello Browser for the bulk of the tools support. Over time, I found a handful of operations that deserved to be easier to use, so I whipped up a handful of OB commands and attached them to the MetacelloProject class.

Select a MetacelloProject class and bring up the Class menu:

Save Packages

Finds the dirty Monticello packages that are members of the selected version of the Metacello project and saves them, prompting you for version name and commit comment. After the packages are saved you are prompted to Update Package Methods.

Update Package Methods

Automatically updates the package spec metadata for the selected version (i.e., modifies and compiles the methods with the #packages:attribute: pragma for the appropriate version to match the currently loaded Monticello package versions). After the methods are update you are prompted to Save Project.

Update Package Repositories

Updates the repository group for each Monitcello package associated with the selected version of the Metacello project to include the repository for that package as specified in the version spec.

[Update 8-26-2009 … more information, if not clearer explanation] When I use Metacello to bootstrap the GemStone Seaside extent, I have TrueRepositories turned off, because I am loading from a disk-based repository (using repositoryOverrides in the loader) so that we don’t _have_ to hit the http repository for every GemStone product build (and for folks using GLASS behind corporate firewalls that don’t have direct access to HTPP repositories). As a result the repositoryGroup is empty for each of the packages. This is a real pain when you start doing development.

‘Update Package Repositories’ scans every package and adds the project repository to the repositoryGroup for each package, so that one doesn’t have to manually do so.

Current Project Version

Displays the current version of the selected Metacello project. The version is calculated by comparing the currently loaded Monticello packages to those specified in the version spec. A leading ‘~’ means that the version is partially loaded (i.e., not all of the packages associated with the project have been loaded into the image).

Load Project Version

Prompts for the version of the selected Metacello project to be loaded. If there are groups associated with the selected version, you are prompted for the group that you would like loaded. When in doubt choose ‘ALL’.

Save Project

Saves the selected Metacello project to the repository specified by the project package of the selected version. You are prompted for version name and commit comment.

Update Project

Loads the latest Monticello package version from the repository specified in the project package of the selected version of the Metacello project. Remember that you are simply loading the Metacello project metadata, so it doesn’t hurt to have the latest metadata loaded. Once the latest version is loaded, you are prompted to Load Project Version.

GLASS Configuration

[Update: 10/14/2009] Note that the information is out-of-date in this section relative to newer versions of Metacello. Please see this post for more up-to-date information.

With Metacello, the configuration of the GLASS appliance goes from this incomprehensible (and pratically uneditable) list of packages:

to a manageable list of Metacello project references:

Best of all, it becomes possible to load a project like Pier into a minimal base image (with only Monticello, Metacllo and OB loaded) using something like the following expression:

(GsPierMetacelloProject version: '1.2') load: { 'Pier Core'. 'Pier AddOns'. }

Not only does Pier get loaded, but all of the projects that Pier depends upon (i.e., Magritte, Seaside, Scriptaculous, etc.) get loaded along with the projects that they depend upon.

For the curious, here are the definitions of the ‘Pier Core’ and ‘Pier AddOns’ groups:

Bootstrapping Metacello

[Update: 10/14/2009] Please see this new section for loading Metacello into an image.

To use Metacello in GLASS, modifications to some base classes are needed, so you will have to wait until I publish a GLASS package to bootstrap Metacello into GLASS.

To bootstrap Metacello into Squeak or Pharo:

1. Load Metacello-All-dkh.10 from http://seaside.gemstone.com/ss/metacello (loads
   Metacello-Core, Metacello-MC, MetacelloProject and OB-Metacello – note that
   Metacello-All needs to have a fairly recent version of OB loaded.
2. Delete the Metacello-All package. It’s only needed to bootstrap Metacello.
3. Execute the following expression in a workspace to load the latest version:

   MetacelloMetacelloProject updateProject.
   MetacelloMetacelloProject latestVersion load.

The #updateProject method performs an Update Project to ensure that you have the most recent Metacello project metadata. The #latestVersion method finds the latest non-development version of Metacello (currently 0.15). The #load method then loads the ‘DEFAULT’ packages for the latest version.

The ‘DEFAULT’ group specifies the bare minimum of packages needed to use Metacello (‘Metacello-Core’, ‘Metacello-MC’, ‘Metacello-Platform’,  and ‘OB-Metacello’).

The ‘Tests’ group specifies the unit test packages for Metacello.

The ‘Samples’ group specifies the package containing some sample Metacello projects including samples for Seaside2.8 and Seaside2.9.

The ‘Tutorial’ group specifies the package containing the Metacello tutorial.

To load the whole Metacello enchilada, you’d execute the following expression:

MetacelloMetacelloProject updateProject.
MetacelloMetacelloProject latestVersion load: 'ALL'.

Tutorial

[Update: 10/14/2009] Please see this new section for loading the Metacello tutorial into an image.

The tutorial covers all of the important Metacello features in some detail. To load the tutorial, execute the following expression:

MetacelloMetacelloProject updateProject.
MetacelloMetacelloProject latestVersion load: 'Tutorial'.

Then open two class  browsers on the MetacelloTutorialProject class. In the left hand browser view the methods in the ‘lessons’ category. In the right hand browser view the ‘–all–‘ category. Read the comments in lesson01 through lesson16OtherExamples.

Current Status

[Update: 10/14/2009] Please see this new section for the current status of the Metacello project.

Version 0.15, the curent version of Metacello, is feature complete, but I’m chasing down infant mortality bugs while I am creating the Metacello projects for GLASS. Once I’m happy, I will release a beta version of Metacello along with the final GLASS package which will boot Metacello into GLASS.

A Metacello Users Guide is also in the works and should be available soon.

—–
[1]Photo by kruemi (Creative Commons).

Deep SIXX with XMLPullParser

Ken Treis has post up called Deep SIXX with XMLPullParser about how he and the folks at his company use SIXX to transfer objects between GemStone and Pharo.

Ken ported Antony Blakey’s XMLPullParser from VisualWorks to Squeak in a (successful) attempt to reduce the amount of memory consumed while copying large object graphs using SIXX.

If you are developing in Squeak/Pharo and deploying in GemStone, then you should check out Ken’s work.

Photo by jayhem (Creative Commons).

MockGemStone Package Available

Ken Treis has started work on a MockGemStone package that provides implementations for a couple of the Rc classes discussed in GemStone 101:Transaction Conflicts. The package can be loaded into a Squeak image and the classes can be used to develop a Seaside application in Squeak that will run in both GemStone and Squeak environments.

The Terse Guide to GemStone Smalltalk

Last week Ramon Leon asked, ‘Where’s the “Terse Guide to Gemstone”?‘ and today I can give him the answer.

In addition to the “Terse Guide to GemStone” you can find links to official GemStone documentation on my docs page.

If, after reading some of the documentation, you find that you have more questions about GemStone, feel free to write comments to this post or you can join the GemStone Mailing list. A number of long-time GemStone users monitor the list (in addition to the GemStone engineering staff) and I’m sure they will be glad to answer your questions.

This week I’ve been ‘heads down’, getting GemSource (the port of SqueakSource to GemStone) ready to go live, so I haven’t had a lot of time to write.

When GemSource goes live next week, it will be a read only site (at first), but we will have published all of the packages (Monticello, Hyper, FastCGI, Seaside, and SqueakSource) for your enjoyment.

Porting Application-Specific Seaside Threads to GemStone

If you are using threads (forking Smalltalk processes) in your Seaside application, then you will need to make some relatively minor mods to your code.

In addition to managing concurrent access to objects (via critical blocks), all threads must be coordinated in relation to transaction boundaries. You cannot afford to have multiple threads aborting and committing without some sort of coordination – the method SeasidePlatformSupport class>>doTransaction: provides that coordination.

SeasidePlatformSupport class>>doTransaction: acquires the transactionMutex, performs a beginTransaction, evalutes the block passed in as an argument, performs a commitTransaction and returns the result of the commitTransaction (a Boolean). The transactionMutex ensures that your transaction will not interfere with any other ongoing transaction, including servicing an HTTP request.

In the SqueakSource Seaside application, there is a CacheThread that runs once a day and the code looks like this:

startCacheThread
  self isCacheThreadRunning ifTrue: [
    self stopCacheTread ].
  cacheThread := [
    self updateCache.
    self updateStatistics.
    (Delay forDuration: 1 day) wait.
  ] forkAt: Processor userBackgroundPriority

After adding a call to SeasidePlatformSupport class>>doTransaction:, the code looks like this:

startCacheThread
  self isCacheThreadRunning ifTrue: [
    self stopCacheTread ].
  cacheThread := [
    SeasidePlatformSupport doTransaction: [
      self updateCache.
      self updateStatistics.
    ].
    (Delay forDuration: 1 day) wait.
  ] forkAt: Processor userBackgroundPriority

If you choose to run the thread in the same VM as the Seaside application, you need to pay attention to how much time you spend inside of a doTransaction: block, because HTTP requests won’t be serviced while the block is executing.

If you’ve got long running transactions, then you should unconditionally move the thread into a separate GemStone VM. In the separate VM, the transaction duration will not affect the response times for HTTP requests.

Here’s an sample topaz script for running this thread in a separate VM (I’ll cover topaz in more detail in another post):

!
! Sample script for updating
!  SSRepository cache and statistics
!
set user DataCurator pass swordfish
login

run
!
! enter manual transaction mode and
!   install SigAbort handler
!
System transactionMode: #manualBegin.
Exception installStaticException:
      [:ex :cat :num :args |
        System abortTransaction.
        System enableSignaledAbortError ]
   category: GemStoneError
   number: 6009
   subtype: nil.
System enableSignaledAbortError.

!
! Start update loop
!
[true] whileTrue:[
   SeasidePlatformSupport
     doTransaction: [
       SSRepository updateCache.
       SSRepository updateStatistics.
   ].
   (Delay forDuration: 1 day) wait.
].
%

If you have more than one thread in your Seaside application, you can fork off additional processes in the script.

My recomendation is that you plan on moving all of your background threads to a separate VM. In the long run it will be simpler to manage your threads and you won’t have to worry nearly as much about how long it takes individual operations to complete.

Transparent Persistence for Seaside

One of the very nice features of GemStone is that you don’t have to do anything special to take advantage of persistence. With other smalltalk dialects (Squeak, VW, VA, and Dolphin) objects that are referenced by the global Smalltalk will be ‘kept alive’ in the image. With GemStone, any objects referenced by a persistent root (I’ll explain why I didn’t say ‘Smalltalk’ here in a later post) will be ‘kept alive’ and ‘automatically’ saved in the data base.

In a generic GemStone application you do have to explicily manage transaction boundaries using the following methods:

• System class>>beginTransaction
• System class>>commitTransaction
• System class>>abortTransaction.

For the GemStone port of Seaside, we have embedded the transaction boundaries in the Seaside framework, so you don’t have to change your application at all – a beginTransaction is performed when the request comes in from the http server and a commitTransaction is performed right before the response is shipped back out to the http server.

This means that while your Seaside application is responding to the request, you have an up-to-date ‘view’ of the object graph and any changes that you make to persistent objects will be committed to the data base, without any special coding on your part.

[Update 3/17/2008]

As you scale your application you will eventually be adding more Seaside vm’s to serve your pages and when you do, you will then need to worry about transaction conflicts. I’ll cover transaction conflict avoidance in a later post.

With recent modifications to the GemStone port of Seaside, it isn’t absolutely neccessary to avoid transaction conflicts. For most applications a Simple Persistence model is perfectly adequate.