Skip to content

Install

Otterdog requires Python 3.11+ to run.

Otterdog tool (command line interface)

System requirements

A few system dependencies are required to be installed:

Mandatory system dependencies

apt install python3 python3-pip pipx
  • poetry: Python package manager
pipx install poetry>=2.0.1
  • git: popular distributed version control system
apt install git

Optional system dependencies

Depending on the type of credential system you are using, install one of the following tools:

snap install bw

apt install pass

Build instructions

After installing the required system dependencies, a virtual python environment needs to be setup and populated with all python dependencies:

$ make init

You should be set to finally run otterdog:

$ ./otterdog.sh --version

Additionally, make init creates a symlink called otterdog in ~/.local/bin, so you can also run it like that:

$ otterdog --version

Shell integration

To enable shell completion, add the following snippet to your shell configuration file (~/.bashrc or ~/.zshrc):

eval "$(_OTTERDOG_COMPLETE=bash_source otterdog)"

eval "$(_OTTERDOG_COMPLETE=zsh_source otterdog)"

When running otterdog in a directory that contains a otterdog.json file, shell completion will be able to suggest organizations found in the otterdog.json file.

Otterdog WebApp

Otterdog WebApp is available as a container image, with releases published to the GitHub Container Registry (GHCR).

For Kubernetes environments, we also provide Helm charts to simplify deployment and configuration.

This guide covers how to deploy the Otterdog WebApp using Helm charts.

System Requirements

Make sure the following requirements are met before deployment:

  • A running Kubernetes cluster
  • Helm installed

GitHub requirements

Create a GitHub App

It is required if you are the integration with GitHub (Create a GitHub app)

  1. Go to your organization https://github.com/organizations/<ORG>
  2. Click Settings.
  3. In the left sidebar, select <> Developer settings
  4. Click GitHub Apps
  5. New GitHub App

Basic Information

  • Add GitHub App name: <choose a name>
  • Homepage URL: <OTTERDOG-WEBAPP>

Webhook

  • [X] Active
  • Webhook url: <OTTERDOG-WEBAPP>/github-webhook/receive
  • Secret: Choose the secret

Add the following permissions and events:

Repository Permissions

Permission Access Level
Actions Read & Write
Administration Read & Write
Codespaces secrets Read & Write
Commit statuses Read & Write
Contents Read & Write
Custom properties Read & Write
Dependabot secrets Read & Write
Environments Read & Write
Issues Read only
Metadata Read only
Pages Read & Write
Pull requests Read & Write
Secrets Read & Write
Variables Read & Write
Webhooks Read & Write
Workflows Read & Write

Organization Permissions

Permission Access Level
Administration Read & Write
Custom Organization Roles Read & Write
Custom properties Admin
Members Read only
Organization codespaces secrets Read & Write
Organization dependabot secrets Read & Write
Plan Read only
Secrets Read & Write
Variables Read & Write
Webhooks Read & Write

The permissions for Codespaces and Dependabot secrets are only used when the default configuration defines the functions that create these secrets, see Dependabot and Codespaces secrets.

Events

Event
Issue comment
Pull request
Pull request review
Push
Workflow job
Workflow run

Note

When permissions are added to an existing GitHub App, each installation keeps its previous permissions until an organization owner accepts the new ones in the organization settings (Settings > GitHub Apps > Configure > Review request).

Deployment Steps

Step 1: Planning the deployment

Before deploying Otterdog, ensure you have the following dependencies ready:

MongoDB: Otterdog requires MongoDB for data persistence. You'll need to deploy MongoDB and have its connection string ready.

Redis or Valkey: Required for caching and message queuing.

  • Important: Valkey/Redis must be configured without authentication as this is not currently supported by GHProxy (included in the Otterdog Helm Charts).

Access controls:

Internal Endpoints Access Control: Otterdog exposes internal management endpoints at https://<address>/internal/.

  • These endpoints should be restricted to Otterdog management only.

  • This can be achieved through ingress configuration using annotations.

    Example:

    ingress:
 annotations:
   nginx.ingress.kubernetes.io/configuration-snippet: |
     location ~* ^/internal/ {
       # Allow only specific IP ranges for management access
       allow 192.168.0.0/16;      # Private network example
       deny all;
     }

Step 2: Add the Helm Repository

helm repo add eclipse-csi https://eclipse-csi.github.io/helm-charts
helm repo update

Step 3: Prepare Configuration

Create a values.yaml file to define your deployment configuration:

This is an example

config:
  configOwner: "<ORGANIZATION>"
  configToken: "<GITHUB TOKEN>"
  configRepo: "otterdog-configs"
  configPath: "otterdog.json"
  dependencyTrackUrl: "https://otterdog-dt.default.svc.cluster"
  dependencyTrackToken: "faketoken"

github:
  webhookSecret: "<WEBHOOK SECRET>"
  appId: "<GITHUB APP ID>"
  appPrivateKey: "<GITHUB APP PRIVATE KEY>"
  webhookValidationContext: "otterdog/otterdog-validation"
  webhookSyncContext: "otterdog/otterdog-sync"

Step 4: Install Otterdog WebApp

helm install eclipse-csi otterdog -f values.yaml

Step 5: Verify the Deployment

Use the following commands to verify that the WebApp is running:

kubectl get pods
kubectl get svc
kubectl get ingress

Init Otterdog

After deployment, Otterdog initializes itself automatically with a helm hook, but this can also be done manually by calling the initialization endpoint::

curl https://<your-otterdog-address>/internal/init

This initialization will: - Fetch the Otterdog configuration from the GitHub repository - Sync the current state from GitHub - Set up the initial data structures in MongoDB - Refresh, in the background, the per organization pull request digests backing the statistics

Note

Refreshing the pull request digests queries GitHub once per organization, so a deployment with many organizations makes the initialization consume a noticeable part of the GitHub rate limit. The refresh stops on its own once the remaining quota gets low, and the statistics then report the organizations it could not read.

The digests are cached in Redis for PULL_REQUEST_STATISTICS_CACHE_TTL seconds, one day by default. Calling /internal/init daily keeps the statistics fresh without anybody ever waiting for GitHub while browsing them.

Note

The initialization refreshes the configuration, pull requests, policies and blueprints of every organization in background tasks. At most MAX_CONCURRENT_INSTALLATION_TASKS of these tasks, 10 by default, run concurrently to bound the memory usage of the webapp and the load on GitHub, so the initialization of a deployment with many organizations takes a while.

The limit applies to each worker process of the webapp, 4 by default, set by the HYPERCORN_WORKERS environment variable. An initialization is handled by a single worker, but refreshes triggered at the same time in different workers, e.g. an initialization and a push to the otterdog configuration repository, run up to this many tasks each.

Validation in the Playground

The Validate button of the Playground posts the evaluated configuration of an organization to /playground/validate and /projects/<project>/playground/validate, which validate it with otterdog, see Validation in the Playground. Like the Playground, these endpoints are public. They neither call GitHub nor store or apply the configuration, but each validation prepares the default configuration and validates the posted one, which takes up to a second for a large organization. They are bounded by two settings:

  • PLAYGROUND_VALIDATION_MAX_SIZE: the maximum size in bytes of a posted configuration, 5 MB by default, a larger one being rejected with the status 413. The evaluated configuration of an organization with 300 repositories takes about 800 KB.
  • PLAYGROUND_MAX_CONCURRENT_VALIDATIONS: the maximum number of validations running concurrently in each worker process of the webapp, 4 by default, the other ones being rejected at once with the status 503.

Note

These settings bound the work of each worker process, not the number of requests of a client. To limit the requests of a client, configure a rate limit for these paths in the ingress or the reverse proxy in front of the webapp.

Updating the Deployment

To upgrade to the latest chart or image version:

helm repo update
helm upgrade eclipse-csi otterdog -f values.yaml

Note

It requires to init Otterdog to sync the PRs

curl https://<your-otterdog-address>/internal/init

Uninstalling Otterdog

To remove the WebApp from your cluster:

helm uninstall otterdog

Additional Resources