Script Handlers

A Sequencer script processes Sequences by defining “handlers” in the script. This is done by completing the special handler functions described below. There are handlers that can be created to process the Setup and Observe commands, which make up the Sequence, but there are also handlers for specific reasons including: aborting and stopping a sequence, putting the Sequencer in Online and Offline modes, and putting the Sequencer into a Diagnostic mode and back to Operations mode. There is also a global error handler to catch all uncaught exceptions, and a shutdown handler to perform cleanup before the Sequencer shut down and exits. Each of these handlers are described below, with a section on how to handle exceptions after that.

Command Handlers

onSetup

This handler is used to handle a Setup command sent to this Sequencer. The handler takes two parameters:

1. command name which is matched against the sequence command sent. If the command name matches, corresponding block provided is executed.
2. block of code which contains logic to act on the Setup command.

In this onSetup example, commands are sent in parallel to each of the WFOS filter wheels.

Kotlin

copysourceonSetup("setupInstrument") {command ->
    // split command and send to downstream
    val assembly1 = Assembly(WFOS, "filter.blueWheel", 5.seconds)
    val assembly2 = Assembly(WFOS, "filter.redWheel", 5.seconds)
    par(
            { assembly1.submit(Setup("WFOS.wfos_darknight", "move")) },
            { assembly2.submit(Setup("WFOS.wfos_darknight", "move")) }
    )
}

In the block provided to this handler, all the CSW services (Event, Alarm, Time Service, etc) and control DSL (loop, par etc) are accessible.

onObserve

This handler is used to handle an Observe command sent to this Sequencer. The handler takes two parameters:

1. command name which is matched against the sequence command sent, if the command name matches, corresponding block provided is executed
2. block of code which contains logic to act on the Observe command.

The following example imagines a WFOS Sequencer receiving an Observe that contains an exposureTime parameter. The exposureTime is extracted into a Setup that is sent to the detector Assembly to start the exposure.

Kotlin

copysource// A detector assembly is defined with a long timeout of 60 minutes
val detectorAssembly = Assembly(WFOS, "detectorAssembly", 60.minutes)
val exposureKey = floatKey("exposureTime")

onObserve("startExposure") { observe ->
    // Extract the input exposure time and send a startObserve command to the detector Assembly
    val expsosureTime = observe(exposureKey).head()
    detectorAssembly.submitAndWait(Setup("WFOS.sequencer", "startObserve", observe.obsId).add(observe(exposureKey)))
}

Online and Offline Handlers

onGoOnline

On receiving the goOnline command, the onGoOnline handler, if defined, will be called. The Sequencer will become online only if the handler executes successfully.

Kotlin

copysourceonGoOnline {
    // send command to downstream components
    assembly.goOnline()
}

onGoOffline

On receiving the goOffline command, the onGoOffline handler, if defined, will be called. The Sequencer will become offline only if the handler executes successfully. Offline handlers could be written to clear the sequencer state before going offline.

Kotlin

copysourceonGoOffline {
    // send command to downstream components
    assembly.goOffline()
}

Abort Sequence Handler

The abort handler could be used to perform any cleanup tasks that need to be done before the current sequence is aborted (e.g. abort an exposure). Note that, even if the handlers fail, the sequence will be aborted.

Kotlin

copysourceonAbortSequence {
    // cleanup steps to be done before aborting will go here
}

Stop Handler

This handler is provided to clear/save the Sequencer state or stop exposures before stopping. Note that, even if the handlers fail, the sequence will be aborted.

Kotlin

copysourceonStop {
    // steps for clearing sequencer-state before stopping will go here
}

Shutdown Handler

This handler will be called just before the Sequencer is shutdown. Note that, even if the handlers fail, the Sequencer will be shutdown.

Kotlin

copysourceonShutdown {
    // cleanup steps to be done before shutdown will go here
}

Diagnostic Mode Handler

This handler can be used to perform actions that need to be done when the Sequencer goes in the diagnostic mode. The handler gets access to two parameters:

• startTime: UTC time at which the diagnostic mode actions should take effect
• hint: represents the diagnostic data mode supported by the Sequencer

The Sequencer can choose to publish any diagnostic data in this handler based on the hint received, and/or send a diagnostic command to downstream components.

Kotlin

copysourcevar diagnosticEventCancellable: Cancellable? = null

onDiagnosticMode { startTime, hint ->
    // start publishing diagnostic data on a supported hint (for e.g. engineering)
    when (hint) {
        "engineering" -> {
            val diagnosticEvent = SystemEvent("ESW.ESW_darknight", "diagnostic")
            diagnosticEventCancellable = schedulePeriodically(startTime, 50.milliseconds) {
                publishEvent(diagnosticEvent)
            }
        }
    }
}

Operations Mode Handler

This handler can be used to perform actions that need to be done when the Sequencer goes in the operations mode. Script writers can use this handler to stop all the publishing being done by the diagnostic mode handler, and/or send an operations mode command to downstream components.

Kotlin

copysourceonOperationsMode {
    // cancel all publishing events done from diagnostic mode
    diagnosticEventCancellable?.cancel()
    // send operations command to downstream
    assembly.operationsMode()
}

Error Handlers

In many cases, any errors encountered in a script would likely cause the command (and therefore, sequence) to fail. Most of the time, not much can be done other than capture and report the error that occurred. It is possible to perform some remediation, but it is likely the sequence would need to run again.

For this reason, we have simplified the error handling of commands such that any DSL APIs that essentially return a negative (e.g. Error or Cancelled) SubmitResponse are recasted as exceptions, which can then be caught by error handlers that are global to the sequence command handler, or the entire script. In this way, such error handling does not need to be repeated throughout the script for each command sent.

A script can error out in following scenarios:

1. Script Initialization Error : When the construction of script throws exception then script initialization fails. In this scenario, the framework will log the error cause. The Sequencer will not start on this failure. One needs to fix the error and then load script again.

2. Command Handlers Failure : While executing a sequence, Command Handlers e.g. onSetup , onObserve can fail because of two reasons:

   1. handler throws exception or
   2. The Command Service or Sequencer Command Service used to interact with downstream Assembly/HCD/Sequencer returns negative SubmitResponse. A negative SubmitResponse is by default considered as error. In this case of failure, sequence is terminated with failure.

3. Handlers Failure : This failure occurs when any of handlers other than Command Handlers fail (e.g. OnGoOnline, onDiagnosticMode etc.). In this scenario, framework will log the error cause. Sequence execution will continue.

The Script DSL provides following constructs to handle failures while executing script:

Global Error Handler

onGlobalError : This construct is provided for the script writer. Logic in the onGlobalError will be executed for all Handler Failures including Command Handler Failures except the Shutdown Handler. If the onGlobalError handler is not provided by script, then only the logging of error cause is done by the framework.

Following example shows usage of onGlobalError

Kotlin

copysource// Scenario-1 onObserve handler fails
onObserve("trigger-filter-wheel") { command ->
    val triggerStartEvent = SystemEvent("esw.command", "trigger.start", command(stringKey(name = "triggerTime")))
    // publishEvent fails with EventServerNotAvailable which fails onObserve handler
    // onGlobalError handler is called
    // Sequence is terminated with failure.
    publishEvent(triggerStartEvent)
}

// Scenario-2 onSetup handler fails - submit returns negative SubmitResponse
onSetup("command-2") { command ->
    val assembly1 = Assembly(IRIS, "filter.wheel", 5.seconds)

    // Submit command to assembly return negative response. (error by default) onGlobalError handler is called.
    // Sequence is terminated with failure.
    assembly1.submit(command)
}

// Scenario-3
onDiagnosticMode { startTime, hint ->
    // publishEvent fails with EventServerNotAvailable
    // onDiagnosticMode handler fails
    // onGlobalError is called. Sequence execution continues.
    publishEvent(SystemEvent("esw.diagnostic.mode", hint))
}

onGlobalError { error ->
    val errorReason = stringKey("reason").set(error.reason)
    val errorEvent = SystemEvent("esw.observation.end", "error", errorReason)
    publishEvent(errorEvent)
}

Error handling at command handler level

onError : This construct is specifically provided for Command Handler Failures. An onError block can be written specifically for each onSetup and onObserve handler. The SubmitResponse error is captured in a ScriptError type and passed to the onError block. This type contains a reason String explaining what went wrong. In case of failure, onError will be called first followed by onGlobalError and the sequence will be terminated with failure. After the error handling blocks are called, the command and sequence, terminate with an Error status.

Kotlin

copysourceonSetup("submit-error-handling") { command ->
    // some logic that results into a Runtime exception
    val result: Int = 1 / 0
}.onError { err ->
    error(err.reason)
}

By default, a negative SubmitResponse is considered an error.

Kotlin

copysourceonSetup("submit-error-handling") { command ->
    val positiveSubmitResponse: SubmitResponse = assembly.submit(command)

}.onError { err ->
    // onError is called when submit command to the assembly fails with a negative response (error, invalid etc)
    error(err.reason)
}

retry: This construct can be attached to an onSetup or onObserve handler to automatically retry the handler code in the case of Command Handler Failures. A retry block expects a retryCount and optional parameter interval which specifies an interval after which onSetup or onObserve will be retried in case of failure. The retry block can be used along with onError or it can be used independently. If retry is combined with onError, the onError block will be called before each retry attempt. If the command handler still fails after all retry attempts, the command fails with an Error status. Then the onGlobalError block will be executed (if provided), and the sequence will be terminated with a failure as well (see Global Error Handler.

The following example shows the retry construct used along with onError.

Kotlin

copysourceonSetup("submit-error-handling") { command ->
    val assembly1 = Assembly(IRIS, "filter.wheel", 5.seconds)

    // Submit command to assembly return negative response. - error by default
    assembly1.submit(command)
}.onError { err ->
    error(err.reason)
}.retry(2)

The following example shows retry with an interval specified and used without an onError block.

Kotlin

copysourceonSetup("submit-error-handling") { command ->
    val assembly1 = Assembly(IRIS, "filter.wheel", 5.seconds)

    // Submit command to assembly return negative response. - error by default
    assembly1.submit(command)
}.retry(2, 10.seconds)