# Woodpecker CI/CD Deployment Architecture ## Goal Replace Drone with Woodpecker while retaining the existing useful property that every service repository is largely self-contained. A new service should not require a corresponding entry in a central deployment repository. At the same time, deployment should no longer give application CI jobs direct access to a Docker socket on a production host. Builds and deployments are separated, production SSH credentials are centralized, and application secrets are stored encrypted in Git using SOPS. ## Components | Component | Responsibility | | --------------------------------- | --------------------------------------------------------------------------------- | | **Gitea** | Git hosting and webhook trigger source | | **Woodpecker CI** | CI pipelines, image builds and triggering deployments | | **Zot** | OCI registry storing container images and immutable deployment artifacts, with retention policies | | **ORAS** | Publishes and retrieves arbitrary deployment bundles as OCI artifacts | | **SOPS + age** | Encrypts application secrets committed to service repositories | | **Central deployment repository** | Contains generic deployment logic and the production credentials | | **Docker Compose** | Defines the runtime configuration of each service | | **Production hosts** | Run the resulting Compose applications, preferably using rootless Docker | | **SSH** | Restricted transport between the central deployment pipeline and production hosts | ## Repository ownership Each application remains responsible for its own deployment definition: ```text passbolt.git ├── Dockerfile ├── Dockerfile.mariadb ├── compose.yaml ├── secrets.prod.env # SOPS-encrypted ├── .sops.yaml └── .woodpecker.yml ``` The repository therefore defines: * container images; * Compose services; * commands and entrypoints; * mounts and volumes; * networks and ports; * non-secret environment variables; * references to secret environment variables; * encrypted production secrets; * the target deployment host or host alias. The central deployment repository contains no per-service definitions: ```text deployments.git ├── .woodpecker/ │ └── deploy.yaml ├── bin/ │ └── deploy-compose └── hosts/ ├── cont1 ├── cont2 └── ... ``` It defines only **how deployments work**, including validation, SOPS decryption, SSH communication and invocation of Docker Compose. ## Deployment workflow ```mermaid flowchart TD Dev["Developer"] -->|push| Gitea["Gitea
Git repositories"] Gitea -->|webhook| WP["Woodpecker CI
service pipeline"] WP --> Test["Tests / quality checks"] Test --> Build["Build container image"] Build -->|push image| Registry["Zot
OCI registry"] WP --> Bundle["Create deployment bundle
compose.yaml
encrypted secrets.prod.env
release metadata"] Bundle -->|ORAS push| Registry Registry --> Image["Container image
@ immutable digest"] Registry --> Artifact["Deployment artifact
@ immutable digest"] WP -->|trigger deployment
artifact digest + target| DeployCI["Woodpecker
deployments.git pipeline"] Artifact -->|ORAS pull| DeployCI SOPSKey["SOPS age private key
Woodpecker secret"] --> DeployCI SSHKey["Production SSH key
Woodpecker secret"] --> DeployCI DeployCI -->|SOPS decrypt
transiently| Runtime["Runtime deployment data
Compose + secrets"] DeployCI -->|restricted SSH| Host["Production host
rootless Docker"] Runtime -->|securely transferred| Host Host --> Pull["docker compose pull"] Pull --> Up["docker compose up -d"] Up --> Running["Running service"] Image -->|pull| Pull ``` ### 1. Build A commit to a service repository triggers its Woodpecker pipeline. Tests and quality checks run first. Woodpecker then builds the container image and publishes it to the OCI registry. The resulting image should ultimately be referenced by an immutable digest rather than only by a mutable tag. The production host does **not** build images. ### 2. Create the deployment artifact The service pipeline packages the deployment-relevant files, for example: ```text deployment/ ├── compose.yaml ├── secrets.prod.env └── release.env ``` `secrets.prod.env` remains SOPS-encrypted at this point. `release.env` can contain non-secret release metadata such as the image reference or commit: ```dotenv IMAGE_TAG=abc123 SOURCE_COMMIT=abc123 ``` The directory is published using ORAS as an OCI artifact: ```text registry.example.org/deploy/passbolt@sha256:... ``` The digest identifies the exact deployment definition and should be treated as the deployable release. This avoids having the deployment pipeline clone or check out the original Git repository. ### 3. Trigger the central deployment pipeline After publishing the artifacts, the application pipeline triggers the Woodpecker pipeline of `deployments.git` using the official `woodpeckerci/plugin-trigger` plugin, which calls the Woodpecker server API to start a pipeline in another repository. Only a small amount of non-secret information needs to be passed as trigger params: ```text ARTIFACT=registry.example.org/deploy/passbolt@sha256:... TARGET=cont2 ``` The trigger step authenticates with a Woodpecker API token stored as a Woodpecker secret. This token should belong to a dedicated bot/service account rather than a personal account, since it grants the ability to start pipelines on any repository the token owner can access, and should be scoped as narrowly as Woodpecker allows. The application pipeline has no production SSH key and no SOPS decryption key. Woodpecker therefore shows the build and deployment as separate pipeline runs, making failures easy to locate and deployments independently inspectable. ### 4. Retrieve and validate the deployment The central deployment pipeline pulls the exact OCI deployment artifact using ORAS. It can validate the artifact before contacting a production machine, including checking the expected structure and Compose configuration. Host names passed by service repositories should be aliases such as: ```text cont1 cont2 cont3 ``` The central deployment machinery resolves these to actual SSH destinations using the corresponding file under `hosts/` in `deployments.git`. Arbitrary SSH hosts, usernames or commands should not be accepted from application repositories; only pre-defined aliases resolve to a target, and an unknown alias fails closed. Initially, resolution is kept simple: any service repository may request deployment to any existing alias, with no per-repository allow-list. Restricting which repositories may deploy to which aliases is a possible later refinement, not required for the initial model. ### 5. Decrypt runtime secrets Production secrets remain encrypted in Git and in the OCI registry. For example, the logical plaintext of `secrets.prod.env` might be: ```dotenv MYSQL_PASSWORD=... MYSQL_ROOT_PASSWORD=... ``` SOPS encrypts these values for multiple age recipients. Expected recipients include: * one personal age key for each DevOps team member; * one dedicated age key for the deployment pipeline; * one offline recovery key. Only the public age recipients are committed to Git. The Woodpecker deployment repository holds the CI age private key as a Woodpecker secret. The central deployment pipeline performs the equivalent of: ```bash sops --decrypt secrets.prod.env ``` Plaintext should exist only transiently during deployment. ### 6. Supply secrets to Docker Compose Existing Compose interpolation can largely remain unchanged: ```yaml services: db: environment: MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD:?err} MYSQL_PASSWORD: ${MYSQL_PASSWORD:?err} passbolt: environment: DATASOURCES_DEFAULT_PASSWORD: ${MYSQL_PASSWORD:?err} ``` The decrypted SOPS dotenv data is supplied to Compose using `--env-file`. Conceptually: ```bash docker compose \ --env-file /temporary/secrets.env \ pull docker compose \ --env-file /temporary/secrets.env \ up -d --remove-orphans ``` The temporary plaintext file should use restrictive permissions and be deleted reliably after the deployment. Longer term, applications supporting `*_FILE` configuration can use Compose secrets instead of exposing credentials as container environment variables, but this is not required for the initial migration. ### 7. Production host The target host receives the Compose deployment through the generic central mechanism and runs: ```bash docker compose pull docker compose up -d --remove-orphans ``` Docker should preferably run rootless under a dedicated deployment/application Unix user or suitable trust domain. The SSH account should ideally be restricted to the deployment operation rather than providing the CI system with an unrestricted interactive shell. ## Secret-management model SOPS makes the service repository self-contained without committing plaintext credentials. For example: ```text passbolt.git ├── compose.yaml ├── secrets.prod.env # encrypted └── .sops.yaml ``` Developers with authorized personal age keys can edit the secrets locally: ```bash sops secrets.prod.env ``` Woodpecker can decrypt the same file using a separate deployment age key. When a DevOps team member joins, their public age recipient is added and the SOPS recipients are updated. When somebody leaves, their recipient is removed and the encrypted data keys are rotated. If an age private key is actually compromised, application credentials contained in historical Git revisions should also be rotated, because old repository history remains decryptable using a compromised old recipient key. ## Registry **Zot** is the chosen OCI registry, run as a small dedicated service rather than reusing Gitea's built-in registry. Zot supports retention policies (e.g. pruning old image and artifact digests), which the Gitea OCI registry does not offer and which is needed to keep registry storage bounded over time. It needs to support two types of content: ```text Container image registry.example.org/fsfe/passbolt@sha256:... Deployment artifact registry.example.org/deploy/passbolt@sha256:... ``` Interoperability with ORAS and arbitrary OCI artifacts should still be verified as part of initial setup, including: * pushing an artifact; * retrieving it; * resolving its digest; * pulling it by digest; * retention/garbage-collection behaviour for untagged or aged-out digests. ## Trust boundaries The intended security model is: ```text Service repository │ ├── may build its application ├── may publish its deployment artifact └── may request deployment to an allowed target │ ▼ Central deployment pipeline │ ├── owns registry pull credentials ├── owns SOPS deployment key ├── owns production SSH credentials ├── validates deployment requests └── performs deployment │ ▼ Production host │ └── rootless Docker / Docker Compose ``` A normal service pipeline therefore never receives: * production SSH private keys; * the SOPS production decryption key; * plaintext secrets belonging to the service; * direct access to a production Docker socket. ## Adding a new service The intended workflow requires no modification of the central deployment repository. A new service needs only: ```text new-service.git ├── Dockerfile ├── compose.yaml ├── secrets.prod.env # when required ├── .sops.yaml └── .woodpecker.yml ``` Its pipeline builds the image, publishes the deployment artifact and triggers the generic deployment pipeline. Central infrastructure needs modification only when the **deployment mechanism or available infrastructure changes**, not when another application is added. # Alternatives considered and dropped ## Direct Docker socket access from CI Current Drone deployments mount a Docker socket from the container host and execute `docker compose up --build -d`. This was rejected because it gives the CI job broad control over the Docker daemon and tightly couples CI, builds and production runtime. Rootless Docker reduces the impact but does not remove the fundamental Docker-socket trust problem. ## Docker over SSH directly from every service pipeline This removes the socket mount and is simple, but each service pipeline would still need credentials providing substantial production access. It was rejected in favour of the central deployment pipeline so production credentials and deployment policy have a single trust boundary. ## Central repository containing every service's Compose configuration This provides a strong centralized source of truth but means every new application requires a second repository change and splits ownership of application deployment configuration. It conflicts with the desired self-service model, so only the **deployment mechanism** is centralized; service definitions remain with their services. ## Deployment pipeline checking out the service Git repository The first central-deployment design cloned the service repository at the exact source commit and retrieved `compose.yaml` from there. While reproducible, this makes Git repository structure and Git transport part of the deployment protocol and feels unnecessarily coupled. It was replaced by immutable OCI deployment artifacts containing exactly the files required for deployment. ## Runtime secrets stored directly on production hosts This would keep runtime secrets completely outside CI and is operationally simple. It was dropped because secrets are currently centrally managed through CI and SOPS provides a cleaner continuation of that model while also allowing authorized DevOps members to manage secrets from the service repository. ## Plain Woodpecker secrets for every application secret Woodpecker can store repository, organization and global secrets, and this would closely resemble the current Drone setup. For dozens of services it would, however, create significant central secret administration and make a generic deployment pipeline difficult because every application has differently named secrets. SOPS avoids this by keeping each service's encrypted secret definitions with that service. ## Bundled Woodpecker secrets Storing an entire application's dotenv file as one Woodpecker secret would simplify generic lookup. This was rejected because bundled secrets are awkward to edit, audit and rotate individually, and provide a poor representation of application configuration. ## OpenBao / Vault-like centralized secret management A dedicated secrets manager would provide powerful policies, dynamic credentials and rotation. It was considered unnecessary infrastructure for the current requirements. SOPS + age provides the required access model without introducing another security-critical service. A dedicated secrets manager remains an option if dynamic credentials or sophisticated automated rotation become necessary. ## Infisical Infisical provides a friendlier centralized secrets-management system with a UI and machine identities. It was not selected because it introduces additional persistent services and operational complexity without currently providing enough benefit over SOPS for this deployment model. ## Native `docker compose publish` Publishing Compose applications directly as OCI artifacts would be attractive. It was not selected because the services rely heavily on host bind mounts, for which native Compose OCI publishing has limitations. Packaging the deployment files with ORAS is simpler and does not need to interpret the Compose definition. ## Harbor Harbor provides extensive OCI registry functionality, including richer access control, robot accounts, scanning, retention and replication. It was considered heavier than necessary. Zot provides the required retention behaviour and ORAS/OCI-artifact support with substantially less operational overhead. ## Gitea OCI/container registry Using Gitea's built-in OCI registry was the initial preference, since it would avoid running another persistent infrastructure service. It was dropped in favour of Zot because Gitea's registry does not support retention policies, which are needed to keep registry storage bounded as images and deployment artifacts accumulate. ORAS/OCI-artifact interoperability with Gitea's registry was also not yet verified at the time this decision was made.