PoC Results

Status: success. The full deployment chain described in concept.md was validated end-to-end and is verified working: a push to a dummy service repository builds two container images, publishes a SOPS-encrypted deployment artifact via ORAS, triggers a separate central deployment pipeline, which pulls the artifact, decrypts the secrets, transfers everything over SSH, and brings up the service via docker compose on a separate target host — with the decrypted secrets confirmed reaching the correct containers and no plaintext secrets left behind afterward.

What was built

Two Hetzner VMs provisioned by Ansible (repo2cicd2deploy, this repo): ci (Woodpecker server+agent, Zot registry, nginx+Let's Encrypt) and target (a plain Docker host acting as "production").
dummy-service (Gitea repo): a two-container dummy service (db + app) with SOPS-encrypted dummy secrets, built and published by its own Woodpecker pipeline.
poc-deployment (Gitea repo): the central deployment repository, containing only generic deployment logic (bin/run-deployment, bin/deploy-compose) and host aliases (hosts/target1), triggered by service repositories via Woodpecker's deployment event.

Verification evidence

docker ps on the target host shows both poc-deploy-db-1 and poc-deploy-app-1 running.
docker logs on both containers confirms the decrypted secrets arrived correctly (DB_ROOT_PASSWORD: received, DB_PASSWORD: received, APP_PASSWORD: received).
No plaintext secrets.decrypted.env remains on the target host after a deployment.
docker logout runs after each deployment; the target host's ~/.docker/config.json shows no lingering registry credentials between runs.
Zot correctly serves both container images and arbitrary ORAS artifacts, retrievable by tag and by digest.
The full trigger chain (dummy-service push → build → publish → trigger → poc-deployment → decrypt → deploy) was exercised repeatedly via real pipeline runs, not just isolated component tests.

Deviations from `concept.md` / `plan.md`

Registry choice was settled before testing Gitea's own OCI registry. concept.md originally planned to test Gitea's built-in registry first and fall back to Zot only if needed. Zot was chosen upfront instead, mainly for its retention-policy support, which Gitea's registry lacks. concept.md was updated during the scoping discussion to reflect this.
The central deployment repository is named poc-deployment, not deployments. This was a naming choice made when the user created the Gitea repository; all references (in dummy-service's pipeline and in the repository's own README) were updated to match.
Deployment logic lives in bin/run-deployment and bin/deploy-compose, not inlined in .woodpecker/deploy.yaml. concept.md's original repository layout already specified a bin/deploy-compose script, which the first implementation draft skipped in favor of inlining everything into the pipeline YAML. This was corrected twice during the PoC (first extracting deploy-compose, then extracting the remaining orchestration logic into run-deployment too) after repeated YAML-quoting issues made the inline approach unworkable. The final deploy.yaml step is under 10 command lines; all real logic is in the two scripts, which are independently testable via SSH without needing a pipeline run.
Restricted SSH, rootless Docker, per-repo host allow-listing and age-key rotation remain deferred, exactly as scoped in plan.md. Nothing in the PoC changed the assessment that these are follow-up hardening items rather than PoC blockers.

Bugs and quirks found (with fixes)

These were discovered by running real pipelines against real infrastructure, not by reading documentation alone. Several are undocumented or under-documented behaviors of Woodpecker and its plugins.

woodpeckerci/plugin-docker-buildx is no longer privileged by default. Needs WOODPECKER_PLUGINS_PRIVILEGED=woodpeckerci/plugin-docker-buildx set on the Woodpecker server, or image builds fail immediately with a linter error.
oras has no Alpine apk package. Any step needing it must download the binary release tarball directly (curl + tar + install).
Woodpecker's environment: block does not support ${VAR} string-substitution. Only settings: blocks and commands: do. Variables meant to be computed from ${CI_COMMIT_SHA:0:8}-style expressions must be exported as the first commands in a step, not set via environment:.
Shell-runtime variables referenced in commands: must be escaped with $$ (e.g. $${REGISTRY}) to prevent Woodpecker's own ${...} pre-processor from evaluating them (and silently substituting empty strings) before the shell ever sees them. Genuine Woodpecker config-scope variables like ${CI_COMMIT_SHA:0:8} are the only ones that should stay single-$.
woodpeckerci/plugin-trigger's repositories: entries require an explicit @branch suffix, even when only using deploy: mode. Without it: build no or branch must be mentioned for deploy.
plugin-trigger's deploy: mode with a non-numeric @branch requires last-successful: true. Without it: for deploy build no must be numeric only or for branch deploy last_successful should be true.
Undocumented: last-successful: true's lookup is hardcoded to match only push-event builds. Confirmed by reading the plugin's Go source directly (impl.go's findFirstBuild, filtering on b.Event == woodpecker.EventPush). It will never match a deployment or manual event build, regardless of the target pipeline's when filters. This is not mentioned anywhere in the plugin's documentation. The fix was to add a dedicated no-op step gated to when: event: push purely so a genuine push-event successful build exists for the lookup to find, alongside the real logic gated to when: event: [deployment, manual].
Woodpecker's "Allow deployments" project setting is off by default and must be manually enabled on the repository being triggered into, or the deploy-trigger API call returns 403. The setting carries an explicit security warning: enabling it lets anyone with push access to that repository use deploy events to reach deploy-scoped secrets.
plugin-trigger's params: setting silently breaks with more than one list item. Multiple KEY=VALUE entries in the YAML list get comma-joined into a single string by Woodpecker's settings-to-environment serialization, but the plugin's CLI argument parser then receives that as one list item rather than splitting it back apart — so the first parameter's value silently absorbs all subsequent parameters, and every parameter after the first is lost. The confirmed-working fix is to write parameters to a small .env-style file first (in its own step) and reference only that single file path in params:, which the plugin reads via godotenv.Read().
A specific Woodpecker command-parsing bug/quirk causes "unterminated quoted string" errors on certain commands: list items containing embedded double-quoted arguments. This is a confirmed real bug in Woodpecker's own command handling (not a real shell syntax error — every affected command was independently verified as valid POSIX shell via sh -n and in a real Alpine container), acknowledged by a Woodpecker maintainer in a public GitHub discussion. Partial fixes (YAML literal block scalars, removing unnecessary quoting) reduced but did not fully eliminate the issue; it also reproduced intermittently on syntactically identical lines across different runs, suggesting a log-streaming or command-dispatch race condition rather than a deterministic parsing bug tied to specific syntax. The most effective fix was architectural: consolidating what had been many separate ssh/scp commands: list items into a single external shell script (bin/run-deployment), invoked from the pipeline with one commands: line. This reduces the number of Woodpecker command-dispatch boundaries to a minimum and has been reliable since.
docker compose pull fails with "no basic auth credentials" if the target host's Docker daemon has never authenticated against the private registry. docker login must run on the target host as part of every deployment (or the credential would otherwise need to persist between runs, which was avoided by adding docker logout after each deployment instead).
nginx's reload signal (SIGHUP) did not reliably pick up newly-added virtual host server blocks in the same Ansible play run that installed nginx fresh. Not fully root-caused; switching the relevant Ansible handler from reload to a full restart resolved it reliably.
The Woodpecker server and agent container images run as a non-root user (UID/GID 1000). Bind-mounted host directories for persistent data must be chowned to match, or the server fails to start with a misleading "no such file or directory" error from its SQLite driver.
oras 1.3.4 fails with 400 Bad Request on monolithic blob uploads over plain HTTP from a workstation over the public internet, because Go's HTTP client sends Transfer-Encoding: chunked instead of Content-Length, which Zot's blob-upload endpoint rejects (confirmed by comparison against a raw curl upload, which succeeds). This did not reproduce when oras was run from the CI host itself over localhost — the actual path real pipelines use — and was also confirmed fixed once TLS termination (nginx + Let's Encrypt) was added in front of Zot. Documented here as a known friction point for local/manual testing against a plain-HTTP registry, not a blocker for the pipeline itself.

Open items deferred beyond this PoC

Unchanged from plan.md's original list, still not needed to validate the core architecture:

Restricted SSH (forced-command / dedicated non-shell user) for the deployment account on production hosts.
Rootless Docker on production hosts.
Per-repository allow-list for which host aliases a service repository may deploy to.
Age key rotation process and procedure for onboarding/offboarding DevOps team members.
A dedicated bot Gitea/Woodpecker account for the cross-repo trigger token (a personal token was used for the PoC).

10 KiB Raw Blame History