mxmehl/repo2cicd2deploy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

add SSH and PoC results

Browse Source
This commit is contained in:
mxmehl
2026-09-18 22:19:25 +02:00
parent 442ddf8816
commit 7c2fa83959
6 changed files with 74 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
POC-RESULTS.md
+54
View File
@@ -0,0 +1,54 @@
# PoC Results
Status: **success**. The full deployment chain described in `concept.md` was validated end-to-end and is verified working: a push to a dummy service repository builds two container images, publishes a SOPS-encrypted deployment artifact via ORAS, triggers a separate central deployment pipeline, which pulls the artifact, decrypts the secrets, transfers everything over SSH, and brings up the service via `docker compose` on a separate target host — with the decrypted secrets confirmed reaching the correct containers and no plaintext secrets left behind afterward.
## What was built
- Two Hetzner VMs provisioned by Ansible (`repo2cicd2deploy`, this repo): `ci` (Woodpecker server+agent, Zot registry, nginx+Let's Encrypt) and `target` (a plain Docker host acting as "production").
- `dummy-service` (Gitea repo): a two-container dummy service (`db` + `app`) with SOPS-encrypted dummy secrets, built and published by its own Woodpecker pipeline.
- `poc-deployment` (Gitea repo): the central deployment repository, containing only generic deployment logic (`bin/run-deployment`, `bin/deploy-compose`) and host aliases (`hosts/target1`), triggered by service repositories via Woodpecker's `deployment` event.
## Verification evidence
- `docker ps` on the target host shows both `poc-deploy-db-1` and `poc-deploy-app-1` running.
- `docker logs` on both containers confirms the decrypted secrets arrived correctly (`DB_ROOT_PASSWORD: received`, `DB_PASSWORD: received`, `APP_PASSWORD: received`).
- No plaintext `secrets.decrypted.env` remains on the target host after a deployment.
- `docker logout` runs after each deployment; the target host's `~/.docker/config.json` shows no lingering registry credentials between runs.
- Zot correctly serves both container images and arbitrary ORAS artifacts, retrievable by tag and by digest.
- The full trigger chain (`dummy-service` push → build → publish → trigger → `poc-deployment` → decrypt → deploy) was exercised repeatedly via real pipeline runs, not just isolated component tests.
## Deviations from `concept.md` / `plan.md`
- **Registry choice was settled before testing Gitea's own OCI registry.** `concept.md` originally planned to test Gitea's built-in registry first and fall back to Zot only if needed. Zot was chosen upfront instead, mainly for its retention-policy support, which Gitea's registry lacks. `concept.md` was updated during the scoping discussion to reflect this.
- **The central deployment repository is named `poc-deployment`, not `deployments`.** This was a naming choice made when the user created the Gitea repository; all references (in `dummy-service`'s pipeline and in the repository's own README) were updated to match.
- **Deployment logic lives in `bin/run-deployment` and `bin/deploy-compose`, not inlined in `.woodpecker/deploy.yaml`.** `concept.md`'s original repository layout already specified a `bin/deploy-compose` script, which the first implementation draft skipped in favor of inlining everything into the pipeline YAML. This was corrected twice during the PoC (first extracting `deploy-compose`, then extracting the remaining orchestration logic into `run-deployment` too) after repeated YAML-quoting issues made the inline approach unworkable. The final `deploy.yaml` step is under 10 command lines; all real logic is in the two scripts, which are independently testable via SSH without needing a pipeline run.
- **Restricted SSH, rootless Docker, per-repo host allow-listing and age-key rotation remain deferred**, exactly as scoped in `plan.md`. Nothing in the PoC changed the assessment that these are follow-up hardening items rather than PoC blockers.
## Bugs and quirks found (with fixes)
These were discovered by running real pipelines against real infrastructure, not by reading documentation alone. Several are undocumented or under-documented behaviors of Woodpecker and its plugins.
1. **`woodpeckerci/plugin-docker-buildx` is no longer privileged by default.** Needs `WOODPECKER_PLUGINS_PRIVILEGED=woodpeckerci/plugin-docker-buildx` set on the Woodpecker server, or image builds fail immediately with a linter error.
2. **`oras` has no Alpine `apk` package.** Any step needing it must download the binary release tarball directly (`curl` + `tar` + `install`).
3. **Woodpecker's `environment:` block does not support `${VAR}` string-substitution.** Only `settings:` blocks and `commands:` do. Variables meant to be computed from `${CI_COMMIT_SHA:0:8}`-style expressions must be `export`ed as the first commands in a step, not set via `environment:`.
4. **Shell-runtime variables referenced in `commands:` must be escaped with `$$` (e.g. `$${REGISTRY}`)** to prevent Woodpecker's own `${...}` pre-processor from evaluating them (and silently substituting empty strings) before the shell ever sees them. Genuine Woodpecker config-scope variables like `${CI_COMMIT_SHA:0:8}` are the only ones that should stay single-`$`.
5. **`woodpeckerci/plugin-trigger`'s `repositories:` entries require an explicit `@branch` suffix**, even when only using `deploy:` mode. Without it: `build no or branch must be mentioned for deploy`.
6. **`plugin-trigger`'s `deploy:` mode with a non-numeric `@branch` requires `last-successful: true`.** Without it: `for deploy build no must be numeric only or for branch deploy last_successful should be true`.
7. **Undocumented: `last-successful: true`'s lookup is hardcoded to match only `push`-event builds.** Confirmed by reading the plugin's Go source directly (`impl.go`'s `findFirstBuild`, filtering on `b.Event == woodpecker.EventPush`). It will never match a `deployment` or `manual` event build, regardless of the target pipeline's `when` filters. This is not mentioned anywhere in the plugin's documentation. The fix was to add a dedicated no-op step gated to `when: event: push` purely so a genuine push-event successful build exists for the lookup to find, alongside the real logic gated to `when: event: [deployment, manual]`.
8. **Woodpecker's "Allow deployments" project setting is off by default** and must be manually enabled on the repository being triggered *into*, or the deploy-trigger API call returns 403. The setting carries an explicit security warning: enabling it lets anyone with push access to that repository use `deploy` events to reach deploy-scoped secrets.
9. **`plugin-trigger`'s `params:` setting silently breaks with more than one list item.** Multiple `KEY=VALUE` entries in the YAML list get comma-joined into a single string by Woodpecker's settings-to-environment serialization, but the plugin's CLI argument parser then receives that as one list item rather than splitting it back apart — so the first parameter's value silently absorbs all subsequent parameters, and every parameter after the first is lost. The confirmed-working fix is to write parameters to a small `.env`-style file first (in its own step) and reference only that single file path in `params:`, which the plugin reads via `godotenv.Read()`.
10. **A specific Woodpecker command-parsing bug/quirk causes "unterminated quoted string" errors** on certain `commands:` list items containing embedded double-quoted arguments. This is a confirmed real bug in Woodpecker's own command handling (not a real shell syntax error — every affected command was independently verified as valid POSIX shell via `sh -n` and in a real Alpine container), acknowledged by a Woodpecker maintainer in a public GitHub discussion. Partial fixes (YAML literal block scalars, removing unnecessary quoting) reduced but did not fully eliminate the issue; it also reproduced intermittently on syntactically identical lines across different runs, suggesting a log-streaming or command-dispatch race condition rather than a deterministic parsing bug tied to specific syntax. The most effective fix was architectural: consolidating what had been many separate `ssh`/`scp` `commands:` list items into a single external shell script (`bin/run-deployment`), invoked from the pipeline with one `commands:` line. This reduces the number of Woodpecker command-dispatch boundaries to a minimum and has been reliable since.
11. **`docker compose pull` fails with "no basic auth credentials"** if the target host's Docker daemon has never authenticated against the private registry. `docker login` must run on the target host as part of every deployment (or the credential would otherwise need to persist between runs, which was avoided by adding `docker logout` after each deployment instead).
12. **`nginx`'s reload signal (SIGHUP) did not reliably pick up newly-added virtual host server blocks** in the same Ansible play run that installed nginx fresh. Not fully root-caused; switching the relevant Ansible handler from `reload` to a full `restart` resolved it reliably.
13. **The Woodpecker server and agent container images run as a non-root user (UID/GID 1000).** Bind-mounted host directories for persistent data must be `chown`ed to match, or the server fails to start with a misleading "no such file or directory" error from its SQLite driver.
14. **`oras` 1.3.4 fails with 400 Bad Request on monolithic blob uploads over plain HTTP from a workstation over the public internet**, because Go's HTTP client sends `Transfer-Encoding: chunked` instead of `Content-Length`, which Zot's blob-upload endpoint rejects (confirmed by comparison against a raw `curl` upload, which succeeds). This did not reproduce when `oras` was run from the CI host itself over `localhost` — the actual path real pipelines use — and was also confirmed fixed once TLS termination (nginx + Let's Encrypt) was added in front of Zot. Documented here as a known friction point for local/manual testing against a plain-HTTP registry, not a blocker for the pipeline itself.
## Open items deferred beyond this PoC
Unchanged from `plan.md`'s original list, still not needed to validate the core architecture:
- Restricted SSH (forced-command / dedicated non-shell user) for the deployment account on production hosts.
- Rootless Docker on production hosts.
- Per-repository allow-list for which host aliases a service repository may deploy to.
- Age key rotation process and procedure for onboarding/offboarding DevOps team members.
- A dedicated bot Gitea/Woodpecker account for the cross-repo trigger token (a personal token was used for the PoC).