add post about docker2caddy

content/blog/2022-04-docker2caddy-reverse-proxy.md

@@ -0,0 +1,155 @@
+---
+title: "Docker2Caddy - An automatic Reverse Proxy for Docker containers"
+date: 2022-04-25
+categories:
+  - english
+tags:
+  - tools
+  - fsfe
+  - server
+headerimage: /blog/docker.jpg
+headercredits: Shipping a load of containers requires a reliable infrastructure
+---
+
+So you have a number of Docker containers running web services which you would
+like to expose to the outside? Well, you probably will at least have considered
+a reverse proxy already. Doing this manually for one, two or even five
+containers may be feasible, but everything above that will be a PITA for sure.
+At the [FSFE](https://fsfe.org) we ran into the same issue with our [own
+distributed container
+infrastructure](https://git.fsfe.org/fsfe-system-hackers/container-server/) at
+and crafted a neat solution that I would like to present to you in the next few
+minutes.
+
+The result is
+[Docker2Caddy](https://git.fsfe.org/fsfe-system-hackers/docker2caddy/) that
+provides a workflow in which you can spin up new containers anytime (e.g. via a
+CI) and the reverse proxy will just do the rest for you magically.
+
+## The assumptions
+
+Let's assume you want to go with reverse proxies to make your web services
+accessible via ports 80 and 443. There are other possibilities, and in more
+complex environments there may be already integrated solutions, but for this
+article we'll wade in a rather simple environment spun up with `docker-compose`[^1].
+
+Let's also assume you care about security and go with a rootless installation of
+Docker. So the daemon will run as an unprivileged user. That's possible but much
+more complex than the default rootful installation[^2]. Because of this, a few
+other solutions will not work, we'll check that later.
+
+Finally, each container shall at least have one separate domain assigned to it
+for which you obviously want to have a valid certificate, e.g. by Let's Encrypt.
+
+In the examples below, we have two containers running, each running a webserver
+listening to port `8080`. The first container shall be available via
+`first.com`, the second via `second.net`. The latter shall also be available via
+`www.second.net`.
+
+[^1]: [This is](https://git.fsfe.org/fsfe-system-hackers/minimal-docker) how a
+    very minimal Docker service in the FSFE infrastructure looks like. For
+    Docker2Caddy, only the `docker-compose.yml` file with its labels is
+    relevant.
+
+[^2]: If you're interested in setting this up via Ansible, I can recommend the
+    [ansible-docker-rootless](https://github.com/konstruktoid/ansible-docker-rootless)
+    role which we integrated in our full-blown
+    [playbook](https://git.fsfe.org/fsfe-system-hackers/container-server/) for
+    the container servers.
+
+## The problems
+
+In the described scenario, there are a number of problem for automating the
+configuration of the reverse proxy in order to direct a domain to the correct
+container, starting with **container discovery** to **IPv6 routing** to
+**handling offline containers**.
+
+The reverse proxy has to be able to discover the currently running containers
+and ideally monitor for changes regularly so that a newly created container with
+a new domain is reachable within a short time without manual intervention.
+
+Before Docker2Caddy we have used
+[nginx-proxy](https://github.com/nginx-proxy/nginx-proxy) combined with
+[acme-companion](https://github.com/nginx-proxy/acme-companion) (formerly known
+as *docker-letsencrypt-nginx-proxy-companion*). These are Docker containers that
+query all containers connected to the `bridge` Docker network. For this to work,
+the containers have to run with environment variables indicating the desired
+domains and local ports that shall be proxied.
+
+In a rootless Docker setup this finally reaches its limits although discovery
+still works. But already before that we did not like the fact that we had to
+connect containers to the bridge network upon creation and therefore lost a bit
+more isolation (which is dubious in Docker anyway).
+
+Now, with rootless, IPv6 was the turning point. Even in rootful Docker setups,
+IPv6 – a 20+ years old, well defined standard protocol – is a pain in the butt.
+But with rootless, the FSFE System Hackers team did not manage to get IPv6
+working in containers to the degree that we needed. While IPv6 traffic reached
+the `nginx-proxy`, it was then treated as IPv4 traffic with the internal Docker
+IP address. That bits you ultimately if you limit requests based on IP
+addresses, e.g. for signups or payments. All traffic via IPv6 will be treated
+as the same internal IPv4 address, therefore triggering the limits regularly.
+
+The easiest solution therefore is to use a reverse proxy running on the host
+system, not as a Docker container with its severe limitations. While the first
+intuition lead us to nginx, we decided to go with
+[Caddy](https://caddyserver.com/). The main advantages we saw are that a virtual
+host in Caddy is very simple to configure and that TLS certificates are
+generated and maintained automatically without extra dependencies like
+`certbot`.
+
+In this setup, containers would need to open their webserver port to the host.
+This "public" port has to be unique per host, but the internal port can stay the
+same, e.g. port `1234` could be mapped to port `8080` inside the container. In
+Caddy you would then configure the domain `first.org` to forward to
+`localhost:1234`. A more or less identical second example container could then
+expose the port `5678` to the host, again listen on `8080` internally, and Caddy
+would redirect `second.net` and `www.second.net` to `localhost:5678`.
+
+But how does Caddy know about the currently running containers and the ports via
+which they want to receive traffic? And how can we handle containers that are
+unavailable, for instance because they crashed or have been deleted for good?
+Docker2Caddy to the rescue!
+
+## The solution
+
+I already concluded that Caddy is a suitable reverse proxy for the outlined use
+case. But in order to be care-free, the configuration has to be generated
+automatically. For this to work, I wrote a rather simple Python application
+called [Docker2Caddy](https://git.fsfe.org/fsfe-system-hackers/docker2caddy)
+that is kept running in the background via a systemd service and writes proper
+logs that are also rotated nicely.
+
+This is how it works internally: it queries (in a configurable interval) the
+Docker daemon for running containers. For each container it looks for specific
+labels (that are also configurable), by default `proxy.host`, `proxy.host_alias`
+and `proxy.port`. If one or multiple containers are found – in our case two –
+one Caddy configuration file per container is created. This is based on a freely
+configurable Jinja2 template. If the configuration changed, e.g. by a new host,
+Caddy will be reloaded and will create a TLS certificate if needed.
+
+But what happens if a container is unavailable? In Docker2Caddy you can
+configure a grace period. Until this is reached, the Caddy configuration for the
+container in question is not removed but could forward to a local or remote
+error page. Only afterwards, the configuration is removed, and Caddy reloaded
+subsequently.
+
+So, what makes Docker2Caddy special? I am biased but see a number of points:
+
+1. **Simplicity**: fundamentally it's a 188 pure lines of code Python script.
+2. **Configurability**: albeit it's simplicity, it's easy to configure for
+   various needs thanks to the templates and the support for rootless Docker
+   setups.
+3. **Adaptability**: it should be rather simple to make Docker2Caddy also work
+   for Podman, or even use different reverse proxies. Feel free to extend it
+   before I'll do it myself someday ;)
+4. **Performance**: while I did not perform before/after benchmarks, Caddy is
+   blazingly fast and will surely perform better on the host than in a limited
+   Docker container.
+
+If you're facing the same challenges in your setup, please feel free to try it
+out. Installation is quite simple and there's even a [minimal Ansible
+playbook](https://git.fsfe.org/fsfe-system-hackers/docker2caddy/src/branch/master/install.yml).
+If you have feedback, I appreciate reading it via comments on Mastodon (see
+below), [email](/contact), or, if you have an FSFE account, as a new issue or
+patch at the [main repo](https://git.fsfe.org/fsfe-system-hackers/docker2caddy).