repo2cicd2deploy/concept.md

Woodpecker CI/CD Deployment Architecture

Goal

Replace Drone with Woodpecker while retaining the existing useful property that every service repository is largely self-contained. A new service should not require a corresponding entry in a central deployment repository.

At the same time, deployment should no longer give application CI jobs direct access to a Docker socket on a production host. Builds and deployments are separated, production SSH credentials are centralized, and application secrets are stored encrypted in Git using SOPS.

Components

Component	Responsibility
Gitea	Git hosting and webhook trigger source
Woodpecker CI	CI pipelines, image builds and triggering deployments
Zot	OCI registry storing container images and immutable deployment artifacts, with retention policies
ORAS	Publishes and retrieves arbitrary deployment bundles as OCI artifacts
SOPS + age	Encrypts application secrets committed to service repositories
Central deployment repository	Contains generic deployment logic and the production credentials
Docker Compose	Defines the runtime configuration of each service
Production hosts	Run the resulting Compose applications, preferably using rootless Docker
SSH	Restricted transport between the central deployment pipeline and production hosts

Repository ownership

Each application remains responsible for its own deployment definition:

passbolt.git
├── Dockerfile
├── Dockerfile.mariadb
├── compose.yaml
├── secrets.prod.env       # SOPS-encrypted
├── .sops.yaml
└── .woodpecker.yml

The repository therefore defines:

• container images;
• Compose services;
• commands and entrypoints;
• mounts and volumes;
• networks and ports;
• non-secret environment variables;
• references to secret environment variables;
• encrypted production secrets;
• the target deployment host or host alias.

The central deployment repository contains no per-service definitions:

deployments.git
├── .woodpecker/
│   └── deploy.yaml
├── bin/
│   └── deploy-compose
└── hosts/
    ├── cont1
    ├── cont2
    └── ...

It defines only how deployments work, including validation, SOPS decryption, SSH communication and invocation of Docker Compose.

Deployment workflow

flowchart TD
    Dev["Developer"] -->|push| Gitea["Gitea<br/>Git repositories"]

    Gitea -->|webhook| WP["Woodpecker CI<br/>service pipeline"]

    WP --> Test["Tests / quality checks"]
    Test --> Build["Build container image"]

    Build -->|push image| Registry["Zot<br/>OCI registry"]

    WP --> Bundle["Create deployment bundle<br/>compose.yaml<br/>encrypted secrets.prod.env<br/>release metadata"]

    Bundle -->|ORAS push| Registry

    Registry --> Image["Container image<br/>@ immutable digest"]
    Registry --> Artifact["Deployment artifact<br/>@ immutable digest"]

    WP -->|trigger deployment<br/>artifact digest + target| DeployCI["Woodpecker<br/>deployments.git pipeline"]

    Artifact -->|ORAS pull| DeployCI

    SOPSKey["SOPS age private key<br/>Woodpecker secret"] --> DeployCI
    SSHKey["Production SSH key<br/>Woodpecker secret"] --> DeployCI

    DeployCI -->|SOPS decrypt<br/>transiently| Runtime["Runtime deployment data<br/>Compose + secrets"]

    DeployCI -->|restricted SSH| Host["Production host<br/>rootless Docker"]

    Runtime -->|securely transferred| Host

    Host --> Pull["docker compose pull"]
    Pull --> Up["docker compose up -d"]
    Up --> Running["Running service"]

    Image -->|pull| Pull

1. Build

A commit to a service repository triggers its Woodpecker pipeline.

Tests and quality checks run first. Woodpecker then builds the container image and publishes it to the OCI registry.

The resulting image should ultimately be referenced by an immutable digest rather than only by a mutable tag.

The production host does not build images.

2. Create the deployment artifact

The service pipeline packages the deployment-relevant files, for example:

deployment/
├── compose.yaml
├── secrets.prod.env
└── release.env

secrets.prod.env remains SOPS-encrypted at this point.

release.env can contain non-secret release metadata such as the image reference or commit:

IMAGE_TAG=abc123
SOURCE_COMMIT=abc123

The directory is published using ORAS as an OCI artifact:

registry.example.org/deploy/passbolt@sha256:...

The digest identifies the exact deployment definition and should be treated as the deployable release.

This avoids having the deployment pipeline clone or check out the original Git repository.

3. Trigger the central deployment pipeline

After publishing the artifacts, the application pipeline triggers the Woodpecker pipeline of deployments.git using the official woodpeckerci/plugin-trigger plugin, which calls the Woodpecker server API to start a pipeline in another repository.

Only a small amount of non-secret information needs to be passed as trigger params:

ARTIFACT=registry.example.org/deploy/passbolt@sha256:...
TARGET=cont2

The trigger step authenticates with a Woodpecker API token stored as a Woodpecker secret. This token should belong to a dedicated bot/service account rather than a personal account, since it grants the ability to start pipelines on any repository the token owner can access, and should be scoped as narrowly as Woodpecker allows.

The application pipeline has no production SSH key and no SOPS decryption key.

Woodpecker therefore shows the build and deployment as separate pipeline runs, making failures easy to locate and deployments independently inspectable.

4. Retrieve and validate the deployment

The central deployment pipeline pulls the exact OCI deployment artifact using ORAS.

It can validate the artifact before contacting a production machine, including checking the expected structure and Compose configuration.

Host names passed by service repositories should be aliases such as:

cont1
cont2
cont3

The central deployment machinery resolves these to actual SSH destinations using the corresponding file under hosts/ in deployments.git. Arbitrary SSH hosts, usernames or commands should not be accepted from application repositories; only pre-defined aliases resolve to a target, and an unknown alias fails closed.

Initially, resolution is kept simple: any service repository may request deployment to any existing alias, with no per-repository allow-list. Restricting which repositories may deploy to which aliases is a possible later refinement, not required for the initial model.

5. Decrypt runtime secrets

Production secrets remain encrypted in Git and in the OCI registry.

For example, the logical plaintext of secrets.prod.env might be:

MYSQL_PASSWORD=...
MYSQL_ROOT_PASSWORD=...

SOPS encrypts these values for multiple age recipients.

Expected recipients include:

• one personal age key for each DevOps team member;
• one dedicated age key for the deployment pipeline;
• one offline recovery key.

Only the public age recipients are committed to Git. The Woodpecker deployment repository holds the CI age private key as a Woodpecker secret.

The central deployment pipeline performs the equivalent of:

sops --decrypt secrets.prod.env

Plaintext should exist only transiently during deployment.

6. Supply secrets to Docker Compose

Existing Compose interpolation can largely remain unchanged:

services:
  db:
    environment:
      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD:?err}
      MYSQL_PASSWORD: ${MYSQL_PASSWORD:?err}

  passbolt:
    environment:
      DATASOURCES_DEFAULT_PASSWORD: ${MYSQL_PASSWORD:?err}

The decrypted SOPS dotenv data is supplied to Compose using --env-file.

Conceptually:

docker compose \
  --env-file /temporary/secrets.env \
  pull

docker compose \
  --env-file /temporary/secrets.env \
  up -d --remove-orphans

The temporary plaintext file should use restrictive permissions and be deleted reliably after the deployment.

Longer term, applications supporting *_FILE configuration can use Compose secrets instead of exposing credentials as container environment variables, but this is not required for the initial migration.

7. Production host

The target host receives the Compose deployment through the generic central mechanism and runs:

docker compose pull
docker compose up -d --remove-orphans

Docker should preferably run rootless under a dedicated deployment/application Unix user or suitable trust domain.

The SSH account should ideally be restricted to the deployment operation rather than providing the CI system with an unrestricted interactive shell.

Secret-management model

SOPS makes the service repository self-contained without committing plaintext credentials.

For example:

passbolt.git
├── compose.yaml
├── secrets.prod.env       # encrypted
└── .sops.yaml

Developers with authorized personal age keys can edit the secrets locally:

sops secrets.prod.env

Woodpecker can decrypt the same file using a separate deployment age key.

When a DevOps team member joins, their public age recipient is added and the SOPS recipients are updated. When somebody leaves, their recipient is removed and the encrypted data keys are rotated.

If an age private key is actually compromised, application credentials contained in historical Git revisions should also be rotated, because old repository history remains decryptable using a compromised old recipient key.

Registry

Zot is the chosen OCI registry, run as a small dedicated service rather than reusing Gitea's built-in registry. Zot supports retention policies (e.g. pruning old image and artifact digests), which the Gitea OCI registry does not offer and which is needed to keep registry storage bounded over time.

It needs to support two types of content:

Container image
registry.example.org/fsfe/passbolt@sha256:...

Deployment artifact
registry.example.org/deploy/passbolt@sha256:...

Interoperability with ORAS and arbitrary OCI artifacts should still be verified as part of initial setup, including:

• pushing an artifact;
• retrieving it;
• resolving its digest;
• pulling it by digest;
• retention/garbage-collection behaviour for untagged or aged-out digests.

Trust boundaries

The intended security model is:

Service repository
    │
    ├── may build its application
    ├── may publish its deployment artifact
    └── may request deployment to an allowed target
            │
            ▼
Central deployment pipeline
    │
    ├── owns registry pull credentials
    ├── owns SOPS deployment key
    ├── owns production SSH credentials
    ├── validates deployment requests
    └── performs deployment
            │
            ▼
Production host
    │
    └── rootless Docker / Docker Compose

A normal service pipeline therefore never receives:

• production SSH private keys;
• the SOPS production decryption key;
• plaintext secrets belonging to the service;
• direct access to a production Docker socket.

Adding a new service

The intended workflow requires no modification of the central deployment repository.

A new service needs only:

new-service.git
├── Dockerfile
├── compose.yaml
├── secrets.prod.env       # when required
├── .sops.yaml
└── .woodpecker.yml

Its pipeline builds the image, publishes the deployment artifact and triggers the generic deployment pipeline.

Central infrastructure needs modification only when the deployment mechanism or available infrastructure changes, not when another application is added.

Alternatives considered and dropped

Direct Docker socket access from CI

Current Drone deployments mount a Docker socket from the container host and execute docker compose up --build -d.

This was rejected because it gives the CI job broad control over the Docker daemon and tightly couples CI, builds and production runtime. Rootless Docker reduces the impact but does not remove the fundamental Docker-socket trust problem.

Docker over SSH directly from every service pipeline

This removes the socket mount and is simple, but each service pipeline would still need credentials providing substantial production access.

It was rejected in favour of the central deployment pipeline so production credentials and deployment policy have a single trust boundary.

Central repository containing every service's Compose configuration

This provides a strong centralized source of truth but means every new application requires a second repository change and splits ownership of application deployment configuration.

It conflicts with the desired self-service model, so only the deployment mechanism is centralized; service definitions remain with their services.

Deployment pipeline checking out the service Git repository

The first central-deployment design cloned the service repository at the exact source commit and retrieved compose.yaml from there.

While reproducible, this makes Git repository structure and Git transport part of the deployment protocol and feels unnecessarily coupled.

It was replaced by immutable OCI deployment artifacts containing exactly the files required for deployment.

Runtime secrets stored directly on production hosts

This would keep runtime secrets completely outside CI and is operationally simple.

It was dropped because secrets are currently centrally managed through CI and SOPS provides a cleaner continuation of that model while also allowing authorized DevOps members to manage secrets from the service repository.

Plain Woodpecker secrets for every application secret

Woodpecker can store repository, organization and global secrets, and this would closely resemble the current Drone setup.

For dozens of services it would, however, create significant central secret administration and make a generic deployment pipeline difficult because every application has differently named secrets.

SOPS avoids this by keeping each service's encrypted secret definitions with that service.

Bundled Woodpecker secrets

Storing an entire application's dotenv file as one Woodpecker secret would simplify generic lookup.

This was rejected because bundled secrets are awkward to edit, audit and rotate individually, and provide a poor representation of application configuration.

OpenBao / Vault-like centralized secret management

A dedicated secrets manager would provide powerful policies, dynamic credentials and rotation.

It was considered unnecessary infrastructure for the current requirements. SOPS + age provides the required access model without introducing another security-critical service. A dedicated secrets manager remains an option if dynamic credentials or sophisticated automated rotation become necessary.

Infisical

Infisical provides a friendlier centralized secrets-management system with a UI and machine identities.

It was not selected because it introduces additional persistent services and operational complexity without currently providing enough benefit over SOPS for this deployment model.

Native docker compose publish

Publishing Compose applications directly as OCI artifacts would be attractive.

It was not selected because the services rely heavily on host bind mounts, for which native Compose OCI publishing has limitations. Packaging the deployment files with ORAS is simpler and does not need to interpret the Compose definition.

Harbor

Harbor provides extensive OCI registry functionality, including richer access control, robot accounts, scanning, retention and replication.

It was considered heavier than necessary. Zot provides the required retention behaviour and ORAS/OCI-artifact support with substantially less operational overhead.

Gitea OCI/container registry

Using Gitea's built-in OCI registry was the initial preference, since it would avoid running another persistent infrastructure service.

It was dropped in favour of Zot because Gitea's registry does not support retention policies, which are needed to keep registry storage bounded as images and deployment artifacts accumulate. ORAS/OCI-artifact interoperability with Gitea's registry was also not yet verified at the time this decision was made.