repo2cicd2deploy/concept.md

# Woodpecker CI/CD Deployment Architecture

## Goal

Replace Drone with Woodpecker while retaining the existing useful property that every service repository is largely self-contained. A new service should not require a corresponding entry in a central deployment repository.

At the same time, deployment should no longer give application CI jobs direct access to a Docker socket on a production host. Builds and deployments are separated, production SSH credentials are centralized, and application secrets are stored encrypted in Git using SOPS.

## Components

| Component                         | Responsibility                                                                    |
| --------------------------------- | --------------------------------------------------------------------------------- |
| **Gitea**                         | Git hosting and, if suitable, OCI registry                                        |
| **Woodpecker CI**                 | CI pipelines, image builds and triggering deployments                             |
| **OCI registry**                  | Stores container images and immutable deployment artifacts                        |
| **ORAS**                          | Publishes and retrieves arbitrary deployment bundles as OCI artifacts             |
| **SOPS + age**                    | Encrypts application secrets committed to service repositories                    |
| **Central deployment repository** | Contains generic deployment logic and the production credentials                  |
| **Docker Compose**                | Defines the runtime configuration of each service                                 |
| **Production hosts**              | Run the resulting Compose applications, preferably using rootless Docker          |
| **SSH**                           | Restricted transport between the central deployment pipeline and production hosts |

## Repository ownership

Each application remains responsible for its own deployment definition:

```text
passbolt.git
├── Dockerfile
├── Dockerfile.mariadb
├── compose.yaml
├── secrets.prod.env       # SOPS-encrypted
├── .sops.yaml
└── .woodpecker.yml
```

The repository therefore defines:

* container images;
* Compose services;
* commands and entrypoints;
* mounts and volumes;
* networks and ports;
* non-secret environment variables;
* references to secret environment variables;
* encrypted production secrets;
* the target deployment host or host alias.

The central deployment repository contains no per-service definitions:

```text
deployments.git
├── .woodpecker/
│   └── deploy.yaml
├── bin/
│   └── deploy-compose
└── hosts/
    ├── cont1
    ├── cont2
    └── ...
```

It defines only **how deployments work**, including validation, SOPS decryption, SSH communication and invocation of Docker Compose.

## Deployment workflow

```mermaid
flowchart TD
    Dev["Developer"] -->|push| Gitea["Gitea<br/>Git repositories"]

    Gitea -->|webhook| WP["Woodpecker CI<br/>service pipeline"]

    WP --> Test["Tests / quality checks"]
    Test --> Build["Build container image"]

    Build -->|push image| Registry["Gitea OCI Registry<br/>or dedicated OCI registry"]

    WP --> Bundle["Create deployment bundle<br/>compose.yaml<br/>encrypted secrets.prod.env<br/>release metadata"]

    Bundle -->|ORAS push| Registry

    Registry --> Image["Container image<br/>@ immutable digest"]
    Registry --> Artifact["Deployment artifact<br/>@ immutable digest"]

    WP -->|trigger deployment<br/>artifact digest + target| DeployCI["Woodpecker<br/>deployments.git pipeline"]

    Artifact -->|ORAS pull| DeployCI

    SOPSKey["SOPS age private key<br/>Woodpecker secret"] --> DeployCI
    SSHKey["Production SSH key<br/>Woodpecker secret"] --> DeployCI

    DeployCI -->|SOPS decrypt<br/>transiently| Runtime["Runtime deployment data<br/>Compose + secrets"]

    DeployCI -->|restricted SSH| Host["Production host<br/>rootless Docker"]

    Runtime -->|securely transferred| Host

    Host --> Pull["docker compose pull"]
    Pull --> Up["docker compose up -d"]
    Up --> Running["Running service"]

    Image -->|pull| Pull
```

### 1. Build

A commit to a service repository triggers its Woodpecker pipeline.

Tests and quality checks run first. Woodpecker then builds the container image and publishes it to the OCI registry.

The resulting image should ultimately be referenced by an immutable digest rather than only by a mutable tag.

The production host does **not** build images.

### 2. Create the deployment artifact

The service pipeline packages the deployment-relevant files, for example:

```text
deployment/
├── compose.yaml
├── secrets.prod.env
└── release.env
```

`secrets.prod.env` remains SOPS-encrypted at this point.

`release.env` can contain non-secret release metadata such as the image reference or commit:

```dotenv
IMAGE_TAG=abc123
SOURCE_COMMIT=abc123
```

The directory is published using ORAS as an OCI artifact:

```text
registry.example.org/deploy/passbolt@sha256:...
```

The digest identifies the exact deployment definition and should be treated as the deployable release.

This avoids having the deployment pipeline clone or check out the original Git repository.

### 3. Trigger the central deployment pipeline

After publishing the artifacts, the application pipeline triggers the Woodpecker pipeline of `deployments.git`.

Only a small amount of non-secret information needs to be passed:

```text
ARTIFACT=registry.example.org/deploy/passbolt@sha256:...
TARGET=cont2
```

The application pipeline has no production SSH key and no SOPS decryption key.

Woodpecker therefore shows the build and deployment as separate pipeline runs, making failures easy to locate and deployments independently inspectable.

### 4. Retrieve and validate the deployment

The central deployment pipeline pulls the exact OCI deployment artifact using ORAS.

It can validate the artifact before contacting a production machine, including checking the expected structure and Compose configuration.

Host names passed by service repositories should be aliases such as:

```text
cont1
cont2
cont3
```

The central deployment machinery resolves these to actual SSH destinations. Arbitrary SSH hosts, usernames or commands should not be accepted from application repositories.

### 5. Decrypt runtime secrets

Production secrets remain encrypted in Git and in the OCI registry.

For example, the logical plaintext of `secrets.prod.env` might be:

```dotenv
MYSQL_PASSWORD=...
MYSQL_ROOT_PASSWORD=...
```

SOPS encrypts these values for multiple age recipients.

Expected recipients include:

* one personal age key for each DevOps team member;
* one dedicated age key for the deployment pipeline;
* one offline recovery key.

Only the public age recipients are committed to Git. The Woodpecker deployment repository holds the CI age private key as a Woodpecker secret.

The central deployment pipeline performs the equivalent of:

```bash
sops --decrypt secrets.prod.env
```

Plaintext should exist only transiently during deployment.

### 6. Supply secrets to Docker Compose

Existing Compose interpolation can largely remain unchanged:

```yaml
services:
  db:
    environment:
      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD:?err}
      MYSQL_PASSWORD: ${MYSQL_PASSWORD:?err}

  passbolt:
    environment:
      DATASOURCES_DEFAULT_PASSWORD: ${MYSQL_PASSWORD:?err}
```

The decrypted SOPS dotenv data is supplied to Compose using `--env-file`.

Conceptually:

```bash
docker compose \
  --env-file /temporary/secrets.env \
  pull

docker compose \
  --env-file /temporary/secrets.env \
  up -d --remove-orphans
```

The temporary plaintext file should use restrictive permissions and be deleted reliably after the deployment.

Longer term, applications supporting `*_FILE` configuration can use Compose secrets instead of exposing credentials as container environment variables, but this is not required for the initial migration.

### 7. Production host

The target host receives the Compose deployment through the generic central mechanism and runs:

```bash
docker compose pull
docker compose up -d --remove-orphans
```

Docker should preferably run rootless under a dedicated deployment/application Unix user or suitable trust domain.

The SSH account should ideally be restricted to the deployment operation rather than providing the CI system with an unrestricted interactive shell.

## Secret-management model

SOPS makes the service repository self-contained without committing plaintext credentials.

For example:

```text
passbolt.git
├── compose.yaml
├── secrets.prod.env       # encrypted
└── .sops.yaml
```

Developers with authorized personal age keys can edit the secrets locally:

```bash
sops secrets.prod.env
```

Woodpecker can decrypt the same file using a separate deployment age key.

When a DevOps team member joins, their public age recipient is added and the SOPS recipients are updated. When somebody leaves, their recipient is removed and the encrypted data keys are rotated.

If an age private key is actually compromised, application credentials contained in historical Git revisions should also be rotated, because old repository history remains decryptable using a compromised old recipient key.

## Registry

The first choice is the existing **Gitea OCI/container registry**, avoiding another persistent infrastructure service.

It needs to support two types of content:

```text
Container image
registry.example.org/fsfe/passbolt@sha256:...

Deployment artifact
registry.example.org/deploy/passbolt@sha256:...
```

Before relying on it, interoperability with ORAS and arbitrary OCI artifacts should be tested, including:

* pushing an artifact;
* retrieving it;
* resolving its digest;
* pulling it by digest.

If Gitea proves unsuitable for arbitrary OCI artifacts, **Zot** is the preferred lightweight dedicated OCI registry.

## Trust boundaries

The intended security model is:

```text
Service repository
    │
    ├── may build its application
    ├── may publish its deployment artifact
    └── may request deployment to an allowed target
            │
            ▼
Central deployment pipeline
    │
    ├── owns registry pull credentials
    ├── owns SOPS deployment key
    ├── owns production SSH credentials
    ├── validates deployment requests
    └── performs deployment
            │
            ▼
Production host
    │
    └── rootless Docker / Docker Compose
```

A normal service pipeline therefore never receives:

* production SSH private keys;
* the SOPS production decryption key;
* plaintext secrets belonging to the service;
* direct access to a production Docker socket.

## Adding a new service

The intended workflow requires no modification of the central deployment repository.

A new service needs only:

```text
new-service.git
├── Dockerfile
├── compose.yaml
├── secrets.prod.env       # when required
├── .sops.yaml
└── .woodpecker.yml
```

Its pipeline builds the image, publishes the deployment artifact and triggers the generic deployment pipeline.

Central infrastructure needs modification only when the **deployment mechanism or available infrastructure changes**, not when another application is added.

# Alternatives considered and dropped

## Direct Docker socket access from CI

Current Drone deployments mount a Docker socket from the container host and execute `docker compose up --build -d`.

This was rejected because it gives the CI job broad control over the Docker daemon and tightly couples CI, builds and production runtime. Rootless Docker reduces the impact but does not remove the fundamental Docker-socket trust problem.

## Docker over SSH directly from every service pipeline

This removes the socket mount and is simple, but each service pipeline would still need credentials providing substantial production access.

It was rejected in favour of the central deployment pipeline so production credentials and deployment policy have a single trust boundary.

## Central repository containing every service's Compose configuration

This provides a strong centralized source of truth but means every new application requires a second repository change and splits ownership of application deployment configuration.

It conflicts with the desired self-service model, so only the **deployment mechanism** is centralized; service definitions remain with their services.

## Deployment pipeline checking out the service Git repository

The first central-deployment design cloned the service repository at the exact source commit and retrieved `compose.yaml` from there.

While reproducible, this makes Git repository structure and Git transport part of the deployment protocol and feels unnecessarily coupled.

It was replaced by immutable OCI deployment artifacts containing exactly the files required for deployment.

## Runtime secrets stored directly on production hosts

This would keep runtime secrets completely outside CI and is operationally simple.

It was dropped because secrets are currently centrally managed through CI and SOPS provides a cleaner continuation of that model while also allowing authorized DevOps members to manage secrets from the service repository.

## Plain Woodpecker secrets for every application secret

Woodpecker can store repository, organization and global secrets, and this would closely resemble the current Drone setup.

For dozens of services it would, however, create significant central secret administration and make a generic deployment pipeline difficult because every application has differently named secrets.

SOPS avoids this by keeping each service's encrypted secret definitions with that service.

## Bundled Woodpecker secrets

Storing an entire application's dotenv file as one Woodpecker secret would simplify generic lookup.

This was rejected because bundled secrets are awkward to edit, audit and rotate individually, and provide a poor representation of application configuration.

## OpenBao / Vault-like centralized secret management

A dedicated secrets manager would provide powerful policies, dynamic credentials and rotation.

It was considered unnecessary infrastructure for the current requirements. SOPS + age provides the required access model without introducing another security-critical service. A dedicated secrets manager remains an option if dynamic credentials or sophisticated automated rotation become necessary.

## Infisical

Infisical provides a friendlier centralized secrets-management system with a UI and machine identities.

It was not selected because it introduces additional persistent services and operational complexity without currently providing enough benefit over SOPS for this deployment model.

## Native `docker compose publish`

Publishing Compose applications directly as OCI artifacts would be attractive.

It was not selected because the services rely heavily on host bind mounts, for which native Compose OCI publishing has limitations. Packaging the deployment files with ORAS is simpler and does not need to interpret the Compose definition.

## Harbor

Harbor provides extensive OCI registry functionality, including richer access control, robot accounts, scanning, retention and replication.

It was considered heavier than necessary. The existing Gitea registry should be tried first; Zot is the preferred lightweight fallback if Gitea's OCI artifact support proves insufficient.