Setup Otterdog¶

After completing the installation of Otterdog, you need to perform some additional steps to configure otterdog.

To start using the cli part of otterdog right away on a specific organization you have to set up the following:

• define a default configuration to use, or use the following default config right away
• create a otterdog.json file that contains a list of GitHub organizations to manager and their respective credentials
• start managing your organizations using the cli

Default configuration¶

The example default config has all supported features enabled and can be used right away. However, reference a released tag or a commit SHA instead of main in base_template, the ref after @ in the example below, to avoid incompatibilities and unreviewed changes. A branch is pulled on every run: every push to it changes the inherited values of your organizations and the validation scripts that otterdog executes with your credentials. Otterdog logs a warning when base_template references a branch.

A change of the default configuration applies to all the organizations using it. Its impact can be evaluated beforehand on the Template impact page of the administration of the otterdog dashboard: the files of the default configuration are edited there, and the configurations of the projects are evaluated with the edited files. The page groups the changes by property across the projects, e.g. repositories[*].delete_branch_on_merge changed from true to false in 807 repositories of 269 projects, and lists the projects whose configuration changes, with their changes, and the ones whose configuration fails to evaluate. The configurations are the ones fetched by the dashboard, and are evaluated in the browser.

Otterdog configuration¶

Create a otterdog.json file with the following content (replace bracketed values according to your setup):

{
  "defaults": {
    "jsonnet": {
      "base_template": "https://github.com/eclipse-csi/otterdog#otterdog/resources/defaults/otterdog-defaults.libsonnet@main",
      "config_dir": "orgs"
    }
  },
  "organizations": [
    {
      "name": "<project-name>",
      "github_id": "<github-id>",
      "credentials": {
        "provider": "plain",
        "api_token": "<GitHub PAT>",
        "username": "<Username>",
        "password": "<Password>",
        "twofa_seed": "<2FA TOTP seed>"
      }
    }
  ]
}

The name of the configuration file can be freely chosen (can be overridden with the -c flag). However, when named otterdog.json, the cli tool will automatically detect and use that file if it is in the current working directory.

Local base template¶

Instead of a file in a GitHub repository, base_template can also be the path to a local file, e.g. to adjust the default configuration for one organization without maintaining a repository for it:

{
  "defaults": {
    "jsonnet": {
      "base_template": "defaults/otterdog-defaults.libsonnet"
    }
  },
  "organizations": [
    {
      "name": "<project-name>",
      "github_id": "<github-id>",
      "base_template": "/path/to/other/otterdog-defaults.libsonnet"
    }
  ]
}

• A relative path is relative to the directory of the otterdog.json file. ~ is expanded.
• The directory of the file is linked as vendor/<directory-name> of the organization, so the configuration of the organization imports it like a template from a repository, e.g. import 'vendor/defaults/otterdog-defaults.libsonnet'. The template can import the other files in its directory, and its hooks are looked up there.
• The link is recreated on every run, changes of the template are picked up right away. It is also done with --local, as there is nothing to fetch.
• The directory of the template must not contain the configuration of the organization, so it cannot be the directory of otterdog.json itself. Keep the template in a directory of its own.
• A value without :// is a path, everything else is the url of a GitHub repository, see above.

Skipping the web UI¶

Some settings can only be read and changed via the GitHub web interface, which requires the username, password and twofa_seed of the credentials. The operations plan, apply, import, show-live, check-status and local-apply skip these settings when called with -n / --no-web-ui.

To skip them by default, e.g. if you only use an access token, set no_web_ui in the defaults of the otterdog.json file:

{
  "defaults": {
    "github": {
      "no_web_ui": true
    }
  }
}

The default can be overridden per call: --web-ui uses the web interface although no_web_ui is set, --no-web-ui skips it although it is not.

Cost policy¶

Pull requests that may incur costs cannot be auto-merged and require a review by the designated team. The thresholds used for this check can be adjusted in the defaults of the otterdog.json file:

{
  "defaults": {
    "cost_policy": {
      "free_max_cache_size_gb": 10
    }
  }
}

Any change of max_cache_size_gb on an existing repository or organization is always considered cost-related.

Credentials¶

Otterdog needs certain credentials to access information from an organization and its repositories on GitHub:

• username / password / 2FA seed
• API token

The login / username / 2FA seed are required to access the web interface of GitHub in order to retrieve certain settings that are not accessible via its rest / graphql API.

The GitHub api token needs to have the following scopes enabled:

• repo
• workflow
• admin:org
• admin:org_hook
• delete_repo

The credentials can be stored in different providers (bitwarden, pass, vault, env).

Bitwarden¶

When using bitwarden to store the credentials, you need to enter a valid item id as additional credential data:

{
  "organizations": [
    {
      "name": "<org name>",
      "github_id": "<github org id>",
      "credentials": {
        "provider": "bitwarden",
        "item_id" : "<bitwarden item id>"
      }
    }
  ]
}

The item stored in bitwarden needs to contain the following information (a sample json output of such an item):

{
  "object": "item",
  "id": "<bitwarden item id>",
  "name": "<item name>",
  "fields": [
    {
      "name": "api_token_admin",
      "value": "<github API token>"
    }
  ],
  "login": {
    "username": "<github username>",
    "password": "<github password>",
    "totp": "<github 2FA TOTP seed>"
  }
}

Mandatory items:

• Field with name "api_token_admin" and as value the GitHub token to access the organization
• login.username of a user that can access the organization with enabled 2FA
• login.password the password of that user
• login.totp the 2FA TOTP seed

Pass¶

When using pass to store the credentials, you need to enter fully qualified pass names to access the various required credential data:

{
  "organizations": [
    {
      "name": "<org name>",
      "github_id": "<github org id>",
      "credentials": {
        "provider": "pass",
        "api_token": "<path/to/api_token>",
        "username": "<path/to/username>",
        "password": "<path/to/password>",
        "twofa_seed": "<path/to/2fa_seed>"
      }
    }
  ]
}

In case your password storage dir is not located at the default location, you can configurate that in the defaults:

{
  "defaults": {
    "pass": {
      "password_store_dir": "path/to/storage/dir"
    }
  }
}

As the password_store_dir might be different on different machines, you can also customize that in a separate .otterdog-defaults.json file:

{
  "pass": {
    "password_store_dir": "path/to/storage/dir"
  }
}

HashiCorp Vault¶

When using a HashiCorp Vault to store the credentials, the secrets are read from a KV version 2 secrets engine. Each credential is referenced as <path/to/secret>/<field>, the last component being the name of the field:

{
  "organizations": [
    {
      "name": "<org name>",
      "github_id": "<github org id>",
      "credentials": {
        "provider": "vault",
        "api_token": "<path/to/secret>/<field>",
        "username": "<path/to/secret>/<field>",
        "password": "<path/to/secret>/<field>",
        "twofa_seed": "<path/to/secret>/<field>"
      }
    }
  ]
}

The connection to the Vault is configured in the defaults:

{
  "defaults": {
    "vault": {
      "vault_addr": "https://vault.example.org",
      "mount_point": "<kv mount point>",
      "verify_ssl": true
    }
  }
}

Environment based credentials¶

When using environment variables to store the credentials, you need to set the following environment variables:

{
  "organizations": [
    {
      "name": "<org name>",
      "github_id": "<github org id>",
      "credentials": {
        "provider": "env",
        "api_token": "<env_var_which_contains_api_token>",
        "username": "<env_var_which_contains_username>",
        "password": "<env_var_which_contains_password>",
        "twofa_seed": "<env_var_which_contains_2fa_seed>"
      }
    }
  ]
}

The environment variables need to be set in the system where otterdog is executed. They can also be set in a .env file in the current working directory, which will be automatically loaded by otterdog.

If you want to keep the same environment variable pattern across organizations, you can set defaults and use placeholders in the defaults.env section. The env provider normalizes placeholder values by converting them to uppercase and replacing spaces and hyphens with underscores. You can use {org_name} and {github_id}.

{
  "defaults": {
    "credentials": {
      "provider": "env"
    },
    "env": {
      "api_token": "OTTER_{org_name}_API_TOKEN",
      "username": "OTTER_{org_name}_USERNAME",
      "password": "OTTER_{github_id}_PASSWORD",
      "twofa_seed": "OTTER_{github_id}_TOTP_SEED"
    }
  }
}

Hint: You can combine this provider with GitHub Actions secrets to securely store the credentials in a GitHub repository and use them in a GitHub action that runs otterdog:

  env:
    OTTER_USERNAME: ${{ vars.OTTER_USERNAME }}
    OTTER_PASSWORD: ${{ secrets.OTTER_PASSWORD }}
    OTTER_TOTP_SEED: ${{ secrets.OTTER_TOTP_SEED }}
    OTTER_API_TOKEN: ${{ secrets.OTTER_API_TOKEN }}

Secret access policy¶

The configuration of an organization references secrets in a credential provider, see Secret Value. This configuration can be changed with pull requests, and otterdog resolves the references with the credentials of the operator, who usually has access to the secrets of all organizations. To prevent an organization from using the secrets of another one, the secrets that an organization can reference can be restricted in the defaults of the otterdog.json file:

{
  "defaults": {
    "secrets": {
      "mode": "enforce",
      "allowed_references": {
        "pass": ["bots/{org_name}/**"],
        "vault": ["{org_name}/**"]
      }
    }
  }
}

Additional references can be allowed for an organization, e.g. secrets shared by several organizations:

{
  "organizations": [
    {
      "name": "<org name>",
      "github_id": "<github org id>",
      "secrets": {
        "allowed_references": {
          "pass": ["bots/shared/sonatype/**"],
          "bitwarden": ["<bitwarden item id>"]
        }
      }
    }
  ]
}

The mode can only be set in the defaults. Without defaults.secrets, only the organizations with secrets have their references restricted, in enforce mode.

The patterns are matched against the path of a reference, segment by segment:

• * matches any part of a segment, e.g. bots/{org_name}/*-token
• ** matches any number of segments, at least one at the end of a pattern: bots/{org_name}/** does not match bots/{org_name} itself

The patterns can use the following placeholders, the examples are given for an organization with the name technology.apoapsis and the github_id eclipse-apoapsis:

The placeholders are replaced by the values of otterdog.json, never by the names in the configuration of the organization, which a pull request can change. Their values are matched literally, e.g. the . of technology.apoapsis is not a wildcard. Any other placeholder is rejected when loading otterdog.json.

A Bitwarden reference <item id>@<field> is matched against its item id, each item has to be allowed explicitly.

A reference must consist of segments separated by / that only contain letters, digits, ., _, @, + and -. Segments . and .. are not allowed, so that a reference can not leave the allowed paths.

The policy is checked when validating a configuration, including the validation of pull requests and the playground of a project in the WebApp, and before resolving each secret during apply.

Environment variables¶

The cli of otterdog reads the following environment variables:

The names of the environment variables that contain the credentials of an organization are defined in the configuration, see Environment based credentials.