Structure¶

Units¶

Nearly all of Ampel’s functionality is implemented in units. A unit is a class that:

Inherits from AmpelBaseModel, and uses annotations to declare the allowed types and default values of is instance variables (which are also keyword arguments to its constructor), and
Is registered in the unit section of the Ampel configuration file.

The configuration of a unit is specified by a UnitModel. When UnitLoader uses a UnitModel to instantiate a unit, it:

Looks up the unit class in the Ampel configuration
Checks that it is a subclass of the requested type
Resolves the configuration if it is an alias, and updates it with the contents of override
Finally, instantiates the class with the entries of the combined configuration dict as keyword arguments.

For some classes, the configuration dict includes more items than are specified directly in the UnitModel:

Subclasses of LogicalUnit have the following additional properties:
- resource, a subset of the resources section of the Ampel configuration. The keys are specified by entries in the class variable require. This can be used to specify the URLs of local services like catalog databases. UnitLoader will raise an exception if no corresponding entry exists in the resources section.
- logger, an instance of AmpelLogger.
All T0, T2, and T3 units are LogicalUnits.
Subclasses of ContextUnit has one additional property:
- context, the complete Ampel configuration, Mongo database connection, etc.
Contributed plugins will not typically provide these.
All subclasses of AmpelBaseModel can have Secret fields:
- If any of the unit’s fields are of type Secret, UnitLoader will look up its value from the configured AbsSecretProvider.
- Use Secret fields for sensitive information like passwords or bearer tokens.
- Secret fields can have a default value of the form {"key": "name_of_secret"}, specifying the name of the secret to use by default. If there is no default, the unit configuration must specify a value.
- When running parts of Ampel manually, you will usually provide the AbsSecretProvider as the secrets argument to AmpelContext, for example via DictSecretProvider.load(). For example, using DevAmpelContext to override some of the configuration parameters from config.yml, and taking secrets from secrets.yaml:
```
from ampel.dev.DevAmpelContext import DevAmpelContext
from ampel.dev.DictSecretProvider import DictSecretProvider
from ampel.model.UnitModel import UnitModel

context = DevAmpelContext.load(
    'config.yml',
    secrets=DictSecretProvider.load('secrets.yaml'),
    db_prefix='AmpelSmokeTest',
    custom_conf = {
        'resource.catsHTM': 'tcp://127.0.0.1:27020',
        'resource.extcats': 'mongodb://localhost:27018',
        'resource.mongo': 'mongodb://localhost:27019',
    }
)

context.loader.secrets.get('name_of_secret')
```

Tiers¶

Data processing is divided into 4 tiers.

Tier 0: Add¶

Ingest (or reject) incoming DataPoints.

Tier 1: Combine¶

Creates Compounds documents, sometimes referred to as ‘states’, based on collections of DataPoints.

Tier 2: Compute¶

Compute derived quantities from newly added StockRecords, DataPoints, and Compounds.

Tier 3: React¶

Perform action based on collections of Ampel objects.