commit d90afe507d3632bdf1a7fcc3b3d350ddf080e5d4 Author: Max Mehl Date: Fri Sep 18 16:45:30 2026 +0200 feat: add concept diff --git a/concept.md b/concept.md new file mode 100644 index 0000000..43e3167 --- /dev/null +++ b/concept.md @@ -0,0 +1,422 @@ +# Woodpecker CI/CD Deployment Architecture + +## Goal + +Replace Drone with Woodpecker while retaining the existing useful property that every service repository is largely self-contained. A new service should not require a corresponding entry in a central deployment repository. + +At the same time, deployment should no longer give application CI jobs direct access to a Docker socket on a production host. Builds and deployments are separated, production SSH credentials are centralized, and application secrets are stored encrypted in Git using SOPS. + +## Components + +| Component | Responsibility | +| --------------------------------- | --------------------------------------------------------------------------------- | +| **Gitea** | Git hosting and, if suitable, OCI registry | +| **Woodpecker CI** | CI pipelines, image builds and triggering deployments | +| **OCI registry** | Stores container images and immutable deployment artifacts | +| **ORAS** | Publishes and retrieves arbitrary deployment bundles as OCI artifacts | +| **SOPS + age** | Encrypts application secrets committed to service repositories | +| **Central deployment repository** | Contains generic deployment logic and the production credentials | +| **Docker Compose** | Defines the runtime configuration of each service | +| **Production hosts** | Run the resulting Compose applications, preferably using rootless Docker | +| **SSH** | Restricted transport between the central deployment pipeline and production hosts | + +## Repository ownership + +Each application remains responsible for its own deployment definition: + +```text +passbolt.git +├── Dockerfile +├── Dockerfile.mariadb +├── compose.yaml +├── secrets.prod.env # SOPS-encrypted +├── .sops.yaml +└── .woodpecker.yml +``` + +The repository therefore defines: + +* container images; +* Compose services; +* commands and entrypoints; +* mounts and volumes; +* networks and ports; +* non-secret environment variables; +* references to secret environment variables; +* encrypted production secrets; +* the target deployment host or host alias. + +The central deployment repository contains no per-service definitions: + +```text +deployments.git +├── .woodpecker/ +│ └── deploy.yaml +├── bin/ +│ └── deploy-compose +└── hosts/ + ├── cont1 + ├── cont2 + └── ... +``` + +It defines only **how deployments work**, including validation, SOPS decryption, SSH communication and invocation of Docker Compose. + +## Deployment workflow + +```mermaid +flowchart TD + Dev["Developer"] -->|push| Gitea["Gitea
Git repositories"] + + Gitea -->|webhook| WP["Woodpecker CI
service pipeline"] + + WP --> Test["Tests / quality checks"] + Test --> Build["Build container image"] + + Build -->|push image| Registry["Gitea OCI Registry
or dedicated OCI registry"] + + WP --> Bundle["Create deployment bundle
compose.yaml
encrypted secrets.prod.env
release metadata"] + + Bundle -->|ORAS push| Registry + + Registry --> Image["Container image
@ immutable digest"] + Registry --> Artifact["Deployment artifact
@ immutable digest"] + + WP -->|trigger deployment
artifact digest + target| DeployCI["Woodpecker
deployments.git pipeline"] + + Artifact -->|ORAS pull| DeployCI + + SOPSKey["SOPS age private key
Woodpecker secret"] --> DeployCI + SSHKey["Production SSH key
Woodpecker secret"] --> DeployCI + + DeployCI -->|SOPS decrypt
transiently| Runtime["Runtime deployment data
Compose + secrets"] + + DeployCI -->|restricted SSH| Host["Production host
rootless Docker"] + + Runtime -->|securely transferred| Host + + Host --> Pull["docker compose pull"] + Pull --> Up["docker compose up -d"] + Up --> Running["Running service"] + + Image -->|pull| Pull +``` + +### 1. Build + +A commit to a service repository triggers its Woodpecker pipeline. + +Tests and quality checks run first. Woodpecker then builds the container image and publishes it to the OCI registry. + +The resulting image should ultimately be referenced by an immutable digest rather than only by a mutable tag. + +The production host does **not** build images. + +### 2. Create the deployment artifact + +The service pipeline packages the deployment-relevant files, for example: + +```text +deployment/ +├── compose.yaml +├── secrets.prod.env +└── release.env +``` + +`secrets.prod.env` remains SOPS-encrypted at this point. + +`release.env` can contain non-secret release metadata such as the image reference or commit: + +```dotenv +IMAGE_TAG=abc123 +SOURCE_COMMIT=abc123 +``` + +The directory is published using ORAS as an OCI artifact: + +```text +registry.example.org/deploy/passbolt@sha256:... +``` + +The digest identifies the exact deployment definition and should be treated as the deployable release. + +This avoids having the deployment pipeline clone or check out the original Git repository. + +### 3. Trigger the central deployment pipeline + +After publishing the artifacts, the application pipeline triggers the Woodpecker pipeline of `deployments.git`. + +Only a small amount of non-secret information needs to be passed: + +```text +ARTIFACT=registry.example.org/deploy/passbolt@sha256:... +TARGET=cont2 +``` + +The application pipeline has no production SSH key and no SOPS decryption key. + +Woodpecker therefore shows the build and deployment as separate pipeline runs, making failures easy to locate and deployments independently inspectable. + +### 4. Retrieve and validate the deployment + +The central deployment pipeline pulls the exact OCI deployment artifact using ORAS. + +It can validate the artifact before contacting a production machine, including checking the expected structure and Compose configuration. + +Host names passed by service repositories should be aliases such as: + +```text +cont1 +cont2 +cont3 +``` + +The central deployment machinery resolves these to actual SSH destinations. Arbitrary SSH hosts, usernames or commands should not be accepted from application repositories. + +### 5. Decrypt runtime secrets + +Production secrets remain encrypted in Git and in the OCI registry. + +For example, the logical plaintext of `secrets.prod.env` might be: + +```dotenv +MYSQL_PASSWORD=... +MYSQL_ROOT_PASSWORD=... +``` + +SOPS encrypts these values for multiple age recipients. + +Expected recipients include: + +* one personal age key for each DevOps team member; +* one dedicated age key for the deployment pipeline; +* one offline recovery key. + +Only the public age recipients are committed to Git. The Woodpecker deployment repository holds the CI age private key as a Woodpecker secret. + +The central deployment pipeline performs the equivalent of: + +```bash +sops --decrypt secrets.prod.env +``` + +Plaintext should exist only transiently during deployment. + +### 6. Supply secrets to Docker Compose + +Existing Compose interpolation can largely remain unchanged: + +```yaml +services: + db: + environment: + MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD:?err} + MYSQL_PASSWORD: ${MYSQL_PASSWORD:?err} + + passbolt: + environment: + DATASOURCES_DEFAULT_PASSWORD: ${MYSQL_PASSWORD:?err} +``` + +The decrypted SOPS dotenv data is supplied to Compose using `--env-file`. + +Conceptually: + +```bash +docker compose \ + --env-file /temporary/secrets.env \ + pull + +docker compose \ + --env-file /temporary/secrets.env \ + up -d --remove-orphans +``` + +The temporary plaintext file should use restrictive permissions and be deleted reliably after the deployment. + +Longer term, applications supporting `*_FILE` configuration can use Compose secrets instead of exposing credentials as container environment variables, but this is not required for the initial migration. + +### 7. Production host + +The target host receives the Compose deployment through the generic central mechanism and runs: + +```bash +docker compose pull +docker compose up -d --remove-orphans +``` + +Docker should preferably run rootless under a dedicated deployment/application Unix user or suitable trust domain. + +The SSH account should ideally be restricted to the deployment operation rather than providing the CI system with an unrestricted interactive shell. + +## Secret-management model + +SOPS makes the service repository self-contained without committing plaintext credentials. + +For example: + +```text +passbolt.git +├── compose.yaml +├── secrets.prod.env # encrypted +└── .sops.yaml +``` + +Developers with authorized personal age keys can edit the secrets locally: + +```bash +sops secrets.prod.env +``` + +Woodpecker can decrypt the same file using a separate deployment age key. + +When a DevOps team member joins, their public age recipient is added and the SOPS recipients are updated. When somebody leaves, their recipient is removed and the encrypted data keys are rotated. + +If an age private key is actually compromised, application credentials contained in historical Git revisions should also be rotated, because old repository history remains decryptable using a compromised old recipient key. + +## Registry + +The first choice is the existing **Gitea OCI/container registry**, avoiding another persistent infrastructure service. + +It needs to support two types of content: + +```text +Container image +registry.example.org/fsfe/passbolt@sha256:... + +Deployment artifact +registry.example.org/deploy/passbolt@sha256:... +``` + +Before relying on it, interoperability with ORAS and arbitrary OCI artifacts should be tested, including: + +* pushing an artifact; +* retrieving it; +* resolving its digest; +* pulling it by digest. + +If Gitea proves unsuitable for arbitrary OCI artifacts, **Zot** is the preferred lightweight dedicated OCI registry. + +## Trust boundaries + +The intended security model is: + +```text +Service repository + │ + ├── may build its application + ├── may publish its deployment artifact + └── may request deployment to an allowed target + │ + ▼ +Central deployment pipeline + │ + ├── owns registry pull credentials + ├── owns SOPS deployment key + ├── owns production SSH credentials + ├── validates deployment requests + └── performs deployment + │ + ▼ +Production host + │ + └── rootless Docker / Docker Compose +``` + +A normal service pipeline therefore never receives: + +* production SSH private keys; +* the SOPS production decryption key; +* plaintext secrets belonging to the service; +* direct access to a production Docker socket. + +## Adding a new service + +The intended workflow requires no modification of the central deployment repository. + +A new service needs only: + +```text +new-service.git +├── Dockerfile +├── compose.yaml +├── secrets.prod.env # when required +├── .sops.yaml +└── .woodpecker.yml +``` + +Its pipeline builds the image, publishes the deployment artifact and triggers the generic deployment pipeline. + +Central infrastructure needs modification only when the **deployment mechanism or available infrastructure changes**, not when another application is added. + +# Alternatives considered and dropped + +## Direct Docker socket access from CI + +Current Drone deployments mount a Docker socket from the container host and execute `docker compose up --build -d`. + +This was rejected because it gives the CI job broad control over the Docker daemon and tightly couples CI, builds and production runtime. Rootless Docker reduces the impact but does not remove the fundamental Docker-socket trust problem. + +## Docker over SSH directly from every service pipeline + +This removes the socket mount and is simple, but each service pipeline would still need credentials providing substantial production access. + +It was rejected in favour of the central deployment pipeline so production credentials and deployment policy have a single trust boundary. + +## Central repository containing every service's Compose configuration + +This provides a strong centralized source of truth but means every new application requires a second repository change and splits ownership of application deployment configuration. + +It conflicts with the desired self-service model, so only the **deployment mechanism** is centralized; service definitions remain with their services. + +## Deployment pipeline checking out the service Git repository + +The first central-deployment design cloned the service repository at the exact source commit and retrieved `compose.yaml` from there. + +While reproducible, this makes Git repository structure and Git transport part of the deployment protocol and feels unnecessarily coupled. + +It was replaced by immutable OCI deployment artifacts containing exactly the files required for deployment. + +## Runtime secrets stored directly on production hosts + +This would keep runtime secrets completely outside CI and is operationally simple. + +It was dropped because secrets are currently centrally managed through CI and SOPS provides a cleaner continuation of that model while also allowing authorized DevOps members to manage secrets from the service repository. + +## Plain Woodpecker secrets for every application secret + +Woodpecker can store repository, organization and global secrets, and this would closely resemble the current Drone setup. + +For dozens of services it would, however, create significant central secret administration and make a generic deployment pipeline difficult because every application has differently named secrets. + +SOPS avoids this by keeping each service's encrypted secret definitions with that service. + +## Bundled Woodpecker secrets + +Storing an entire application's dotenv file as one Woodpecker secret would simplify generic lookup. + +This was rejected because bundled secrets are awkward to edit, audit and rotate individually, and provide a poor representation of application configuration. + +## OpenBao / Vault-like centralized secret management + +A dedicated secrets manager would provide powerful policies, dynamic credentials and rotation. + +It was considered unnecessary infrastructure for the current requirements. SOPS + age provides the required access model without introducing another security-critical service. A dedicated secrets manager remains an option if dynamic credentials or sophisticated automated rotation become necessary. + +## Infisical + +Infisical provides a friendlier centralized secrets-management system with a UI and machine identities. + +It was not selected because it introduces additional persistent services and operational complexity without currently providing enough benefit over SOPS for this deployment model. + +## Native `docker compose publish` + +Publishing Compose applications directly as OCI artifacts would be attractive. + +It was not selected because the services rely heavily on host bind mounts, for which native Compose OCI publishing has limitations. Packaging the deployment files with ORAS is simpler and does not need to interpret the Compose definition. + +## Harbor + +Harbor provides extensive OCI registry functionality, including richer access control, robot accounts, scanning, retention and replication. + +It was considered heavier than necessary. The existing Gitea registry should be tried first; Zot is the preferred lightweight fallback if Gitea's OCI artifact support proves insufficient.