Oscilloscope[1]

One of the harder problems that developers have to deal with is characterizing issues that show up only in production, especially if exceptions aren’t being thrown.

In development, you’d just toss a `self halt` or two into the code and you’d be off to the races, but in production where multiple servers are involved, things can get a little more complicated. Often developers must resort to print statement debugging and sophisticated server log mining tools to characterize production issues.

Remote Breakpoints to the Rescue

Breakpoints have been available in GLASS, for a long time now. In 2008 I added remote breakpoints for Seaside2.8 and finally with the release of Seaside 3.0.6.3, remote breakpoints are available for Seaside 3.0.

With remote breakpoints, you can easily set/clear breakpoints in all of the gems that are serving Seaside and when a breakpoint is encountered a continuation is snapped off and added to the ObjectLog. At your leisure, you can debug the continuation from GemTools or inspect the continuation via the ObjectLog.

Step by Step Guide to Remote Breakpoints for Seaside 3.0

  1. Setup for Remote Breakpoints:
    1. GemStone/S 2.x
    2. GemStone/S 3.x
  2. Launch Seaside
  3. Testing, Testing…:
    1. Set Remote Breakpoint
    2. Trigger Breakpoint with WARemoteDebuggingWalkbackErrorHandler
    3. Trigger Breakpoint with WAGemStoneProductionErrorHandler
    4. Debug Breakpoint in GemTools
    5. View Breakpoint in ObjectLog
    6. Clear Remote Breakpoints

Setup for Remote Breakpoints

Before using remote breakpoints, you need to do a little prep depending upon which version of GemStone/S you are using.

  1. GemStone/S 2.x
  2. GemStone/S 3.x

Remote Breakpoint Setup for GemStone/S 2.x

If you want to use remote breakpoints with GemStone/S 2.x, you need only install a new StartSeaside30 Adaptor script or add the following section to your own script:

true "enable for remote breakpoints and profiling"
  ifTrue: [
    GemToGemAnnouncement installStaticHandler.
    Exception
      installStaticException: [:ex :cat :num :args |
          BreakpointNotification signal.
          "needed to avoid infinite loop when resuming
           from a breakpoint"
          ex _incrementBreakpointsToIgnore. ]
      category: GemStoneError
      number: 6005
      subtype: nil.
   System commitTransaction
      ifFalse: [
         nil error: 'Could not commit for GemToGemSignaling' ]].

Remote Breakpoint Setup for GemStone/S 3.x

The script support for remote breakpoints is preinstalled in GemStone/S 3.0.1, however you must disable native code by setting GEM_NATIVE_CODE_ENABLED to FALSE in your gem conf files:

GEM_NATIVE_CODE_ENABLED = FALSE;

In 3.x there are actually three conf files to think about:

  1. $GEMSTONE/seaside/data/system.conf
  2. $GEMSTONE/seaside/etc/seaside30.conf
  3. $GEMSTONE/seaside/etc/maintenance30.conf

By default, GEM_NATIVE_CODE_ENABLED is set to FALSE, but in a production installation you should set GEM_NATIVE_CODE_ENABLED to TRUE to get the best performance and only set it to FALSE if you want to enable remote breakpoints.

Launch Seaside

  1. Start up a GemTools development image.
  2. Specify the adaptor and number of Seaside server gems to be used (note that the Seaside30 session workspace also has the necessary template expressions).
  3. Start the Seaside server gems.

Testing, Testing…

Lets run through the process of using remote breakpoints:

  1. Set Remote Breakpoint
  2. Trigger Breakpoint with WARemoteDebuggingWalkbackErrorHandler
  3. Trigger Breakpoint with WAGemStoneProductionErrorHandler
  4. Debug Breakpoint in GemTools
  5. View Breakpoint in ObjectLog
  6. Clear Remote Breakpoints

For this example we’ll set a breakpoint in the WAExceptionFunctionalTest>>raiseWarning method.

Set Remote Breakpoint

  1. If you’ve just started your Seaside gems for this test, then you should force a transaction by selecting either the ‘Commit’ or ‘Abort’ menu item on the GemTools Transaction menu. The list of running Seaside gems is maintained in a persistent list and the action for starting the gems from GemTools doesn’t involve a transaction. If there is no commit or abort between the time you’ve started the Seaside gems and set the breakpoint, the breakpoint will not be set in the Seaside gems.
  2. In the GemTools development image, bring up a class editor on the WAExceptionFunctionalTest>>raiseWarning method, place the cursor at the beginning of the #notify selector and select the ‘set breakpoint’ menu item:

Trigger Breakpoint with WARemoteDebuggingWalkbackErrorHandler

  1. Bring up a web browser on the WAFunctionalTest page:
    http://localhost:8383/tests/functional/WAExceptionFunctionalTest
  2. Ensure that you are using either the WARemoteDebuggingWalkbackErrorHandler or WAGemStoneProductionErrorHandler by selecting the correct class from the error handler drop down list (don’t forget to click on the ‘Set Handler’ button):
  3. Trigger the breakpoint by clicking on the ‘Raise warning’ link:
  4. The Seaside Walkback page will come up:
  5. Click on the ‘Remote Debug’ link to snap off a continuation add it to the ObjectLog:
  6. The remote debug page will come up, confirming that the breakpoint continuation has been added to the ObjectLog:

Trigger Breakpoint with WAGemStoneProductionErrorHandler

  1. Go back to theWAFunctionalTest page and select the WAGemStoneProductionErrorHandler this time:
  2. When you trigger the breakpoint clicking on the ‘Raise warning’ link, instead of a walkback window you get the production error message window for the Warning that is signaled as part of the functional test:
  3. This pages is what you’d expect without the breakpoint, but since you are using the WAGemStoneProductionErrorHandler, the breakpoint continuation is snapped off and silently added to the ObjectLog along with a continuation for the Warning exception. We can check that by looking at the Debug menu in GemTools:

Debug Breakpoint in GemTools

  1. In GemTools use the Debug menu to select the breakpoint continuation:
  2. and bring up the debugger on the continuation:

View Breakpoint in ObjectLog

You can inspect continuations in the ObjectLog from your web browser by using the WAObjectLog component.

  1. Register the component by following the instructions on the ObjectLog page in the wiki.
  2. Hit http://localhost:8383/tools/objectLog in your web browser to bring up the Object Log page:
  3. Scan the list and find the item labeled ‘continuation’ and click on the ‘2’ link:
  4. When you click on the ‘2’ link you are taken to an object inspector on the ObjectLogEntry itself. From there you can navigate to the objects stashed in the entry:
  5. To view a printString of the stack click on the ‘ GsProcess(oop=206376961, status=debug, priority=15, WARemoteDebuggingWalkbackErrorHandler >> open:’ link:
  6. This page doesn’t provide as much information as the debugger view, but sometimes a stack trace is enough:

Clearing Remote Breakpoints

  1. To clear the breakpoints, you can bring up the text menu and select ‘clear ALL breakpoints’ to clear all of the breakpoints that may be set or ‘clear method breakpoints’ to clear the breakpoints that are set in the selected method:

—–
[1]http://www.flickr.com/photos/mwichary/2356597072 / CC BY 2.0