156 lines
8.1 KiB
Markdown
156 lines
8.1 KiB
Markdown
---
|
||
title: "Docker2Caddy - An automatic Reverse Proxy for Docker containers"
|
||
date: 2022-04-25
|
||
categories:
|
||
- english
|
||
tags:
|
||
- tools
|
||
- fsfe
|
||
- server
|
||
headerimage: /blog/docker.jpg
|
||
headercredits: Shipping a load of containers requires a reliable infrastructure
|
||
---
|
||
|
||
So you have a number of Docker containers running web services which you would
|
||
like to expose to the outside? Well, you probably will at least have considered
|
||
a reverse proxy already. Doing this manually for one, two or even five
|
||
containers may be feasible, but everything above that will be a PITA for sure.
|
||
At the [FSFE](https://fsfe.org) we ran into the same issue with our [own
|
||
distributed container
|
||
infrastructure](https://git.fsfe.org/fsfe-system-hackers/container-server/) at
|
||
and crafted a neat solution that I would like to present to you in the next few
|
||
minutes.
|
||
|
||
The result is
|
||
[Docker2Caddy](https://git.fsfe.org/fsfe-system-hackers/docker2caddy/) that
|
||
provides a workflow in which you can spin up new containers anytime (e.g. via a
|
||
CI) and the reverse proxy will just do the rest for you magically.
|
||
|
||
## The assumptions
|
||
|
||
Let's assume you want to go with reverse proxies to make your web services
|
||
accessible via ports 80 and 443. There are other possibilities, and in more
|
||
complex environments there may be already integrated solutions, but for this
|
||
article we'll wade in a rather simple environment spun up with `docker-compose`[^1].
|
||
|
||
Let's also assume you care about security and go with a rootless installation of
|
||
Docker. So the daemon will run as an unprivileged user. That's possible but much
|
||
more complex than the default rootful installation[^2]. Because of this, a few
|
||
other solutions will not work, we'll check that later.
|
||
|
||
Finally, each container shall at least have one separate domain assigned to it
|
||
for which you obviously want to have a valid certificate, e.g. by Let's Encrypt.
|
||
|
||
In the examples below, we have two containers running, each running a webserver
|
||
listening to port `8080`. The first container shall be available via
|
||
`first.com`, the second via `second.net`. The latter shall also be available via
|
||
`www.second.net`.
|
||
|
||
[^1]: [This is](https://git.fsfe.org/fsfe-system-hackers/minimal-docker) how a
|
||
very minimal Docker service in the FSFE infrastructure looks like. For
|
||
Docker2Caddy, only the `docker-compose.yml` file with its labels is
|
||
relevant.
|
||
|
||
[^2]: If you're interested in setting this up via Ansible, I can recommend the
|
||
[ansible-docker-rootless](https://github.com/konstruktoid/ansible-docker-rootless)
|
||
role which we integrated in our full-blown
|
||
[playbook](https://git.fsfe.org/fsfe-system-hackers/container-server/) for
|
||
the container servers.
|
||
|
||
## The problems
|
||
|
||
In the described scenario, there are a number of problem for automating the
|
||
configuration of the reverse proxy in order to direct a domain to the correct
|
||
container, starting with **container discovery** to **IPv6 routing** to
|
||
**handling offline containers**.
|
||
|
||
The reverse proxy has to be able to discover the currently running containers
|
||
and ideally monitor for changes regularly so that a newly created container with
|
||
a new domain is reachable within a short time without manual intervention.
|
||
|
||
Before Docker2Caddy we have used
|
||
[nginx-proxy](https://github.com/nginx-proxy/nginx-proxy) combined with
|
||
[acme-companion](https://github.com/nginx-proxy/acme-companion) (formerly known
|
||
as *docker-letsencrypt-nginx-proxy-companion*). These are Docker containers that
|
||
query all containers connected to the `bridge` Docker network. For this to work,
|
||
the containers have to run with environment variables indicating the desired
|
||
domains and local ports that shall be proxied.
|
||
|
||
In a rootless Docker setup this finally reaches its limits although discovery
|
||
still works. But already before that we did not like the fact that we had to
|
||
connect containers to the bridge network upon creation and therefore lost a bit
|
||
more isolation (which is dubious in Docker anyway).
|
||
|
||
Now, with rootless, IPv6 was the turning point. Even in rootful Docker setups,
|
||
IPv6 – a 20+ years old, well defined standard protocol – is a pain in the butt.
|
||
But with rootless, the FSFE System Hackers team did not manage to get IPv6
|
||
working in containers to the degree that we needed. While IPv6 traffic reached
|
||
the `nginx-proxy`, it was then treated as IPv4 traffic with the internal Docker
|
||
IP address. That bits you ultimately if you limit requests based on IP
|
||
addresses, e.g. for signups or payments. All traffic via IPv6 will be treated
|
||
as the same internal IPv4 address, therefore triggering the limits regularly.
|
||
|
||
The easiest solution therefore is to use a reverse proxy running on the host
|
||
system, not as a Docker container with its severe limitations. While the first
|
||
intuition lead us to nginx, we decided to go with
|
||
[Caddy](https://caddyserver.com/). The main advantages we saw are that a virtual
|
||
host in Caddy is very simple to configure and that TLS certificates are
|
||
generated and maintained automatically without extra dependencies like
|
||
`certbot`.
|
||
|
||
In this setup, containers would need to open their webserver port to the host.
|
||
This "public" port has to be unique per host, but the internal port can stay the
|
||
same, e.g. port `1234` could be mapped to port `8080` inside the container. In
|
||
Caddy you would then configure the domain `first.org` to forward to
|
||
`localhost:1234`. A more or less identical second example container could then
|
||
expose the port `5678` to the host, again listen on `8080` internally, and Caddy
|
||
would redirect `second.net` and `www.second.net` to `localhost:5678`.
|
||
|
||
But how does Caddy know about the currently running containers and the ports via
|
||
which they want to receive traffic? And how can we handle containers that are
|
||
unavailable, for instance because they crashed or have been deleted for good?
|
||
Docker2Caddy to the rescue!
|
||
|
||
## The solution
|
||
|
||
I already concluded that Caddy is a suitable reverse proxy for the outlined use
|
||
case. But in order to be care-free, the configuration has to be generated
|
||
automatically. For this to work, I wrote a rather simple Python application
|
||
called [Docker2Caddy](https://git.fsfe.org/fsfe-system-hackers/docker2caddy)
|
||
that is kept running in the background via a systemd service and writes proper
|
||
logs that are also rotated nicely.
|
||
|
||
This is how it works internally: it queries (in a configurable interval) the
|
||
Docker daemon for running containers. For each container it looks for specific
|
||
labels (that are also configurable), by default `proxy.host`, `proxy.host_alias`
|
||
and `proxy.port`. If one or multiple containers are found – in our case two –
|
||
one Caddy configuration file per container is created. This is based on a freely
|
||
configurable Jinja2 template. If the configuration changed, e.g. by a new host,
|
||
Caddy will be reloaded and will create a TLS certificate if needed.
|
||
|
||
But what happens if a container is unavailable? In Docker2Caddy you can
|
||
configure a grace period. Until this is reached, the Caddy configuration for the
|
||
container in question is not removed but could forward to a local or remote
|
||
error page. Only afterwards, the configuration is removed, and Caddy reloaded
|
||
subsequently.
|
||
|
||
So, what makes Docker2Caddy special? I am biased but see a number of points:
|
||
|
||
1. **Simplicity**: fundamentally it's a 188 pure lines of code Python script.
|
||
2. **Configurability**: albeit it's simplicity, it's easy to configure for
|
||
various needs thanks to the templates and the support for rootless Docker
|
||
setups.
|
||
3. **Adaptability**: it should be rather simple to make Docker2Caddy also work
|
||
for Podman, or even use different reverse proxies. Feel free to extend it
|
||
before I'll do it myself someday ;)
|
||
4. **Performance**: while I did not perform before/after benchmarks, Caddy is
|
||
blazingly fast and will surely perform better on the host than in a limited
|
||
Docker container.
|
||
|
||
If you're facing the same challenges in your setup, please feel free to try it
|
||
out. Installation is quite simple and there's even a [minimal Ansible
|
||
playbook](https://git.fsfe.org/fsfe-system-hackers/docker2caddy/src/branch/master/install.yml).
|
||
If you have feedback, I appreciate reading it via comments on Mastodon (see
|
||
below), [email](/contact), or, if you have an FSFE account, as a new issue or
|
||
patch at the [main repo](https://git.fsfe.org/fsfe-system-hackers/docker2caddy).
|