Example Configuration

What does a configuration file look like?

This section contains examples of alto configuration files. These examples are simply meant to help familiarize users. This is the exact configuration file that is generated when you run alto init and all represent exactly the same thing. The choice of format is down to user preference though we recommend toml. Once you understand the general structure, the next few sections will dive into the details of each field.

TOML

TOML

The advantage of toml is that it is a bit more compact than yaml. It is a great choice for smaller projects. It is also the only format that can colocate nested keys. This is useful for putting environment-specific config in the same place.

alto.toml

# Alto Starter Project

# Anything in the default section is applied to all environments
# you can have an arbitrary number of environments. The ALTO_ENV
# environment variable is used to determine which environment to
# layer over the default section.
[default]
project_name = "{project}"
# Enable builtin extensions or bring your own
extensions = ["evidence"]
load_path = "raw"
# Set env vars that will be available to all plugins
environment.STARTER_PROJECT = 1

[default.taps.tap-carbon-intensity] # https://gitlab.com/meltano/tap-carbon-intensity
pip_url = "git+https://gitlab.com/meltano/tap-carbon-intensity.git#egg=tap_carbon_intensity"
load_path = "carbon_intensity"
capabilities = ["state", "catalog"]
select = [
  "*.*",
  # PII hashing
  "~*.dnoregion"
]

[default.taps.tap-bls] # https://hub.meltano.com/extractors/tap-bls
pip_url = "git+https://github.com/frasermarlow/tap-bls#egg=tap_bls"
capabilities = ["state", "catalog"]
load_path = "bls"
select = ["JTU000000000000000JOR", "JTU000000000000000JOL"]
# Configuring the tap
config.startyear = "2019"
config.endyear = "2020"
config.calculations = "true"
config.annualaverage = "false"
config.aspects = "false"
config.disable_collection = "true"
config.update_state = "false"
config.series_list_file_location = "./series.json"

[default.targets.target-jsonl] # https://hub.meltano.com/loaders/target-singer-jsonl
pip_url = "target-jsonl==0.1.4"
# this.load_path is a reference to the load_path of the tap
config.destination_path = "@format output/{this.load_path}"

[default.utilities.dlt] # https://github.com/dlt-hub/dlt
pip_url = "python-dlt[duckdb]>=0.2.0a25"
environment.PEX_INHERIT_PATH = "fallback"

# Example of a custom environment
# This environment will be layered over the default section
# and will override any values that are the same in both sections.
# Dicts are merged, lists are appended. This would be active if
# the environment variable ALTO_ENV=github_actions
[github_actions]
load_path = "cicd"
targets.target-jsonl.config.destination_path = "@format /github/workspace/output/{this.load_path}"

YAML

YAML

The advantage of using yaml is that it is the most ubiquitous format. Many users will already be familiar with it. While toml has the advantage of not requiring indentation, yaml's indentation can prove a strength given larger configuration files.

alto.yaml

# Alto Starter Project

# Anything in the default section is applied to all environments
# you can have an arbitrary number of environments. The ALTO_ENV
# environment variable is used to determine which environment to
# layer over the default section.
default:
  project_name: "{project}"
  # Enable builtin extensions or bring your own
  extensions:
    - evidence
  load_path: raw
  environment:
    # Set env vars that will be available to all plugins
    STARTER_PROJECT: 1
  taps:
    tap-carbon-intensity: # https://gitlab.com/meltano/tap-carbon-intensity
      pip_url: "git+https://gitlab.com/meltano/tap-carbon-intensity.git#egg=tap_carbon_intensity"
      load_path: carbon_intensity
      capabilities:
        - state
        - catalog
      select:
        - "*.*"
        # PII hashing
        - "~*.dnoregion"
      config: {}
    tap-bls: # https://hub.meltano.com/extractors/tap-bls
      pip_url: "git+https://github.com/frasermarlow/tap-bls#egg=tap_bls"
      capabilities:
        - state
        - catalog
      load_path: bls
      select:
        - JTU000000000000000JOR
        - JTU000000000000000JOL
      config:
        startyear: "2019"
        endyear: "2020"
        calculations: "true"
        annualaverage: "false"
        aspects: "false"
        disable_collection: "true"
        update_state: "false"
        series_list_file_location: ./series.json
  targets:
    target-jsonl: # https://hub.meltano.com/loaders/target-singer-jsonl
      pip_url: target-jsonl==0.1.4
      config:
        # this.load_path is a reference to the load_path of the tap
        destination_path: "@format output/{this.load_path}"
  utilities:
    dlt:
      pip_url: "python-dlt[duckdb]>=0.2.0a25"
      environment:
        # Set plugin specific env vars
        PEX_INHERIT_PATH: fallback

# Example of a custom environment
# This environment will be layered over the default section
# and will override any values that are the same in both sections.
# Dicts are merged, lists are appended. This would be active if
# the environment variable ALTO_ENV=github_actions
github_actions:
  load_path: cicd
  targets:
    target-jsonl:
      config:
        destination_path: "@format /github/workspace/output/{this.load_path}"

JSON

JSON

The primary value in json is that is extremely common and easy to generate programmatically. Most languages can do this without third party libs. The alto dump command also produces json which alto iself can use.

alto.json

{
    "default": {
        "project_name": "{project}",
        "extensions": [
            "evidence"
        ],
        "load_path": "raw",
        "environment": {
            "ALTO_STARTER_PROJECT": 1
        },
        "taps": {
            "tap-carbon-intensity": {
                "pip_url": "git+https://gitlab.com/meltano/tap-carbon-intensity.git#egg=tap_carbon_intensity",
                "load_path": "carbon_intensity",
                "capabilities": [
                    "state",
                    "catalog"
                ],
                "select": [
                    "*.*",
                    "~*.dnoregion"
                ],
                "config": { }
            },
            "tap-bls": {
                "pip_url": "git+https://github.com/frasermarlow/tap-bls#egg=tap_bls",
                "capabilities": [
                    "state",
                    "catalog"
                ],
                "load_path": "bls",
                "select": [
                    "JTU000000000000000JOR",
                    "JTU000000000000000JOL"
                ],
                "config": {
                    "startyear": "2019",
                    "endyear": "2020",
                    "calculations": "true",
                    "annualaverage": "false",
                    "aspects": "false",
                    "disable_collection": "true",
                    "update_state": "false",
                    "series_list_file_location": "./series.json"
                }
            }
        },
        "targets": {
            "target-jsonl": {
                "pip_url": "target-jsonl==0.1.4",
                "config": {
                    "destination_path": "@format output/{this.load_path}"
                }
            }
        },
        "utilities": {
            "dlt": {
                "pip_url": "python-dlt[duckdb]>=0.2.0a25",
                "environment": {
                    "PEX_INHERIT_PATH": "fallback"
                }
            }
        }
    },
    "github_actions": {
        "load_path": "cicd",
        "targets": {
            "target-jsonl": {
                "config": {
                    "destination_path": "@format /github/workspace/output/{this.load_path}"
                }
            }
        }
    }
}