Structure¶
Tiers¶
Data processing is divided into 4 tiers.
Tier 0: Add¶
Ingest (or reject) incoming DataPoints
.
Tier 1: Combine¶
Creates Compounds
documents, sometimes referred to as ‘states’, based on collections of DataPoints
.
Tier 2: Compute¶
Compute derived quantities from newly added StockRecords
, DataPoints
, and Compounds
.
Tier 3: React¶
Perform action based on collections of Ampel objects.
The default top level implementation provided by ampel-core
is the class T3Processor
, which is capable of spawning and executing
T3 processes according to the provided configuration.
The execution typically follows a 5-step process specified by the following fields of T3Directive
:
Note
Within Ampel, a unique ID called ‘stock ID’ is assigned to all documents associated with a given entity across T1, T2 and T3 (and possibly T0 if already known at this point).
context¶
Package |
|
Governing abstract class |
|
Known implementations |
|
Allows to generate global information that will be provided to T3 units. Note that the information gathered in this stage is not associated with individual ampel elements (each identified by a unique ‘stock ID’).
select¶
Package |
|
Governing abstract class |
|
Known implementations |
|
Allows to select which elements should be provided to T3 units.
The default implementation T3StockSelector
selects stock IDs based on
criteria targeting the internal collection ‘stock’.
Note that other implementations are possible, in particular implementations
based on the information from the internal collection ‘t2’.
The returned sequence of stock IDs is passed to the next stage.
load¶
Package |
|
Governing abstract class |
|
Known implementations |
|
Regulates which documents to load for each ampel ID selected by the previous stage.
The loaded documents are then included into AmpelBuffer
instances which are passed to the next stages.
Note that all current implementations rely internally on the backend class DBContentLoader
.
complement¶
Package |
|
Governing abstract class |
|
Known implementations |
|
Allows to include additional information to the loaded AmpelBuffer
instances.
Most implementations are expected to update AmpelBuffer.extra
which was specifically added for this purpose.
Other implementations, such as T3ExtJournalAppender
update the content
of the AMPEL core documents themselve.
Since the information only lives in the process memory
(not saved into the DB, only included into the views),
there is no hard limitation wrt to data type / serialization property / size
run¶
The last stage executes t3 units according the provided configuration and using the various information loaded from previous stages.
The default implementation T3UnitRunner
provided by ampel-core
follows a 3-step process governed by the following sections from the configuration:
filter
project
execute
filter¶
Package |
|
Governing abstract class |
|
Known implementations |
|
Note
This setting applies only in case the underlying T3 process runs multiple units.
The optional setting ‘filter’ allows to define selection critera for the loaded AmpelBuffer
instances.
Only matching instances are passed to the next stage.
Example: say you want to select all new entities that were associated with your channel the last 24 hours, and post information about them to slack. Furthermore, say your entities come in blue or red and you’d like to post “blue entities” into the slack channel “#blue” and “red entities” into the “#red” channel. 1) Either you define two separate processes with distinct top level setting select, which will then be scheduled seperately. You do not need the setting filter in this case. 2) Or you can create a single process selecting both red and blue entities and define two “run blocks” under the top-level setting run. The first run-block sub-selects blue entities and posts them to “#blue” and the second one sub-selects red entities and posts then to “#red”.
project¶
Package |
|
Governing abstract class |
|
Known implementations |
|
The process potentially:
strips out ‘channel’ attributes
removes journal entries not associated with configured channels
removes t2 results not associated with configured channels
Note
this stage, although modular as most of Ampel is, is not expected to be customized in most cases.
Note
the template associated with channel definitions usually automatically configure this stage
execute¶
Package |
|
Governing abstract class |
|
Known implementations |
|
Last stage during which T3 units are instantiated and AmpelBuffer
instances
converted into views (e.g. SnapView
and subclasses, containing pseudo-immutable structures).
T3 unit instances (i.e. instances of AbsT3Unit
) are provided both with views and
global information loaded during the previous stages.
They have also the possibility to customize the journal entry
created each time the underlying process is run.