repo2cicd2deploy/plan.md

PoC Implementation Plan

This plan validates the architecture described in concept.md end-to-end before adopting it for real services. It was drafted after a scoping discussion; the decisions below reflect that discussion.

Problem statement

Validate the concept.md architecture (Gitea push → Woodpecker build → Zot registry via ORAS → cross-pipeline trigger → central deployments.git pipeline → SSH → docker compose up on a target host) end-to-end with a dummy multi-container service, before adopting it for real services.

Scope decisions

• Full end-to-end flow is in scope, not just isolated pieces (e.g. just the registry).
• Two Hetzner VMs, cheapest available tier, hostnames to be provided by the user once created:
  • ci host: Woodpecker (server + agent) + Zot, rootful Docker.
  • target host: the "production" host, rootful Docker.
• Ansible in this repo (repo2cicd2deploy) provisions both hosts: one inventory, host groups ci and target, single playbook run.
• New repos to be created outside this repo, in the user's self-hosted Gitea:
  • deployments.git — central deploy pipeline.
  • a dummy service repo — DB container with a root password plus an app container that receives the DB password and its own separate password. All values are dummy/fake, never real secrets, and are SOPS-encrypted regardless.
• Registry: Zot, with htpasswd basic auth (simplest viable auth for a PoC) and a retention policy configured.
• Cross-pipeline trigger: official woodpeckerci/plugin-trigger plugin, authenticated with a dedicated bot Gitea account's Woodpecker API token (not a personal token).
• Host alias resolution: kept simple, lives in deployments.git, no per-repo allow-list for this PoC.
• SSH: normal key-based SSH, no forced-command restriction (explicitly deferred, not needed for PoC).
• Dummy service: two containers (db + app), no real application code. The app container only needs to prove it received the correct env vars, e.g. by logging env | grep -E 'DB_PASSWORD|APP_PASSWORD' on startup and then staying up.
• Rootless Docker is explicitly deferred; the PoC uses rootful Docker on both hosts.

Background / research findings

• Woodpecker has its own secret store, separate from Gitea. Gitea Actions secrets are unrelated and unused here; Gitea is only the Git forge and webhook source.
• Cross-repo pipeline triggering is supported via the official woodpeckerci/plugin-trigger plugin, which calls the Woodpecker REST API with a token, can pass arbitrary params, and can trigger a deploy-type pipeline event.
• Zot supports OCI artifacts (ORAS-compatible), unlike Gitea's built-in registry, and supports retention policies; htpasswd is the simplest auth backend for a PoC.
• Gitea↔Woodpecker integration is a standard OAuth application registration in Gitea (client ID/secret) plus WOODPECKER_GITEA_URL / WOODPECKER_GITEA_CLIENT / WOODPECKER_GITEA_SECRET environment variables on the Woodpecker server.
• concept.md was updated during the scoping discussion to reflect all of the above (Zot as the chosen registry, host-alias resolution living in deployments.git, the concrete plugin-trigger mechanism with a bot-token note).

Architecture for the PoC

flowchart TD
    subgraph CI["ci host (Hetzner VM 1)"]
        WPS["Woodpecker server + agent"]
        Zot["Zot registry"]
    end
    subgraph TGT["target host (Hetzner VM 2)"]
        Compose["docker compose<br/>dummy-db + dummy-app"]
    end
    Gitea["existing Gitea"] -->|webhook| WPS
    Dev["dummy-service.git push"] --> Gitea
    WPS -->|build image, push| Zot
    WPS -->|ORAS push artifact| Zot
    WPS -->|plugin-trigger| WPS
    WPS -->|ORAS pull artifact, SOPS decrypt, SSH| TGT

Both VMs are provisioned by one Ansible playbook (site.yml) with two groups: ci and target. Roles: docker (both groups), zot and woodpecker (ci group only).

Task breakdown

Task 1: Ansible skeleton and Docker role

Set up inventory/hosts.yml with ci and target groups (placeholder hostnames until provided), a docker role installing Docker CE (rootful) via the official apt repository, and a site.yml playbook applying it to both groups.

• Test: ansible-playbook site.yml --check runs without errors against a real inventory entry; docker run hello-world succeeds on both hosts after a real run.
• Demo: both VMs have working rootful Docker, verified with docker version over an Ansible ad-hoc command.

Task 2: Zot role and registry smoke test

Add a zot role deploying Zot via Docker Compose on the ci host, with htpasswd basic auth and a retention policy configured. Generate credentials and store the htpasswd file as an Ansible-vault-encrypted variable, or generate it idempotently on the host.

• Test: from a workstation, oras login, then oras push/oras pull a trivial test artifact (a text file) against the Zot instance, and docker push/docker pull a trivial image — confirms Zot handles both image and arbitrary-artifact OCI content as the concept requires.
• Demo: a manually pushed test artifact and test image are both retrievable by digest from Zot.

Task 3: Woodpecker server/agent role and Gitea integration

Add a woodpecker role deploying Woodpecker server + agent via Docker Compose on the ci host. Register a Gitea OAuth application for Woodpecker (documented manual step, since it requires the Gitea UI — note this in a README rather than automating it). Wire WOODPECKER_GITEA_URL / WOODPECKER_GITEA_CLIENT / WOODPECKER_GITEA_SECRET.

• Test: log into the Woodpecker UI via Gitea SSO, activate a throwaway test repo, confirm a trivial .woodpecker.yml (e.g. echo hello) runs successfully on push.
• Demo: a test push to a scratch repo triggers a visible, successful Woodpecker pipeline run.

Task 4: Dummy service repo — build and publish

Create dummy-service.git in Gitea with:

• Dockerfile.db (e.g. FROM alpine, sets up a fake "DB" that just accepts a root password env var)

• Dockerfile.app (e.g. FROM alpine, entrypoint script env | grep -E 'DB_PASSWORD|APP_PASSWORD'; sleep infinity)

• compose.yaml referencing ${IMAGE_TAG}-style placeholders for both images

• secrets.prod.env (dummy DB_ROOT_PASSWORD, DB_PASSWORD, APP_PASSWORD) encrypted with SOPS/age (a fresh age keypair generated for this PoC only)

• .sops.yaml and .woodpecker.yml building both images, pushing them to Zot, and bundling and pushing the deployment artifact (compose.yaml + encrypted secrets.prod.env + release.env) via ORAS

• Test: push triggers Woodpecker; verify both images and the deployment artifact land in Zot with resolvable digests (oras discover/crane digest or equivalent).

• Demo: a git push to dummy-service.git results in two images and one deployment artifact visible in Zot.

Task 5: Central deployments repo — trigger wiring

Create deployments.git with .woodpecker/deploy.yaml reacting to a deployment event, and hosts/target1 containing the SSH connection details (hostname, user) for the target VM. Add the plugin-trigger step to dummy-service.git's pipeline (Task 4), authenticated with a dedicated bot account's Woodpecker token stored as a Woodpecker secret on dummy-service.git.

• Test: push to dummy-service.git; confirm the deployments.git pipeline is triggered with the correct ARTIFACT and TARGET params visible in its build log.
• Demo: one push to the service repo visibly triggers a second, separate pipeline run in deployments.git with the right parameters.

Task 6: Central deployments repo — decrypt, transfer, deploy

Extend deploy.yaml to: resolve TARGET against hosts/, ORAS-pull the artifact by the given digest, SOPS-decrypt secrets.prod.env using the deployment age key (stored as a repo-level Woodpecker secret on deployments.git), transfer the compose file and decrypted env to the target host, then SSH-run docker compose pull && docker compose up -d --remove-orphans, then delete the remote plaintext secrets file.

• Test: end-to-end run from a dummy-service.git push results in both containers running on the target host; inspecting the app container (exec or logs) shows the correct decrypted DB_PASSWORD/APP_PASSWORD values reaching it, proving the full secrets path.
• Demo: full pipeline, push-to-running-containers, verified by inspecting the app container's logged env vars on the target host.

Task 7: Wrap-up documentation

Add a short POC-RESULTS.md (or similar) in this repo capturing what worked, any deviations from concept.md, and open items to resolve before using this for a real service (e.g. restricted SSH, per-repo host allow-list, key rotation process). Update concept.md if the PoC surfaces anything materially different from what is documented there.

• Demo: a short written record for future reference to decide next steps — no new code, just documentation.

Open items deferred beyond this PoC

• Restricted SSH (forced-command / dedicated non-shell user) for the deployment account on production hosts.
• Rootless Docker on production hosts.
• Per-repository allow-list for which host aliases a service repository may deploy to.
• Age key rotation process and procedure for onboarding/offboarding DevOps team members.