Structure

Tiers

Data processing is divided into 4 tiers.

Tier 0: Add

Ingest (or reject) incoming DataPoints.

Tier 1: Combine

Creates Compounds documents, sometimes referred to as ‘states’, based on collections of DataPoints.

Tier 2: Compute

Compute derived quantities from newly added StockRecords, DataPoints, and Compounds.

Tier 3: React

Perform action based on collections of Ampel objects.

The default top level implementation provided by ampel-core is the class T3Processor, which is capable of spawning and executing T3 processes according to the provided configuration. The execution typically follows a 5-step process specified by the following fields of T3Directive:

Note

Within Ampel, a unique ID called ‘stock ID’ is assigned to all documents associated with a given entity across T1, T2 and T3 (and possibly T0 if already known at this point).

context

Package

ampel.t3.context

Governing abstract class

AbsT3RunContextAppender

Known implementations

T3AddLastRunTime, T3AddAlertsNumber

Allows to generate global information that will be provided to T3 units. Note that the information gathered in this stage is not associated with individual ampel elements (each identified by a unique ‘stock ID’).

select

Package

ampel.t3.select

Governing abstract class

AbsT3Selector

Known implementations

T3StockSelector, T3FilteringStockSelector

Allows to select which elements should be provided to T3 units. The default implementation T3StockSelector selects stock IDs based on criteria targeting the internal collection ‘stock’. Note that other implementations are possible, in particular implementations based on the information from the internal collection ‘t2’. The returned sequence of stock IDs is passed to the next stage.

load

Package

ampel.t3.load

Governing abstract class

AbsT3Loader

Known implementations

T3SimpleDataLoader, T3LatestStateDataLoader

Regulates which documents to load for each ampel ID selected by the previous stage. The loaded documents are then included into AmpelBuffer instances which are passed to the next stages. Note that all current implementations rely internally on the backend class DBContentLoader.

complement

Package

ampel.t3.complement

Governing abstract class

AbsT3DataAppender

Known implementations

T3ExtJournalAppender, SEDMSpectrumAppender, TNSNames, ZTFCutoutImages

Allows to include additional information to the loaded AmpelBuffer instances. Most implementations are expected to update AmpelBuffer.extra which was specifically added for this purpose. Other implementations, such as T3ExtJournalAppender update the content of the AMPEL core documents themselve. Since the information only lives in the process memory (not saved into the DB, only included into the views), there is no hard limitation wrt to data type / serialization property / size

run

The last stage executes t3 units according the provided configuration and using the various information loaded from previous stages.

The default implementation T3UnitRunner provided by ampel-core follows a 3-step process governed by the following sections from the configuration:

  • filter

  • project

  • execute

filter

Package

ampel.t3.run.filter

Governing abstract class

AbsT3Filter

Known implementations

T3AmpelBufferFilter

Note

This setting applies only in case the underlying T3 process runs multiple units.

The optional setting ‘filter’ allows to define selection critera for the loaded AmpelBuffer instances. Only matching instances are passed to the next stage.

Example: say you want to select all new entities that were associated with your channel the last 24 hours, and post information about them to slack. Furthermore, say your entities come in blue or red and you’d like to post “blue entities” into the slack channel “#blue” and “red entities” into the “#red” channel. 1) Either you define two separate processes with distinct top level setting select, which will then be scheduled seperately. You do not need the setting filter in this case. 2) Or you can create a single process selecting both red and blue entities and define two “run blocks” under the top-level setting run. The first run-block sub-selects blue entities and posts them to “#blue” and the second one sub-selects red entities and posts then to “#red”.

project

Package

ampel.t3.run.project

Governing abstract class

AbsT3Projector

Known implementations

T3BaseProjector, T3ChannelProjector

The process potentially:

  • strips out ‘channel’ attributes

  • removes journal entries not associated with configured channels

  • removes t2 results not associated with configured channels

Note

this stage, although modular as most of Ampel is, is not expected to be customized in most cases.

Note

the template associated with channel definitions usually automatically configure this stage

execute

Package

ampel.t3.run

Governing abstract class

AbsT3UnitRunner

Known implementations

T3UnitRunner, T3DynamicUnitRunner

Last stage during which T3 units are instantiated and AmpelBuffer instances converted into views (e.g. SnapView and subclasses, containing pseudo-immutable structures).

T3 unit instances (i.e. instances of AbsT3Unit) are provided both with views and global information loaded during the previous stages. They have also the possibility to customize the journal entry created each time the underlying process is run.