|
|
|
|
@ -0,0 +1,155 @@
|
|
|
|
|
---
|
|
|
|
|
title: "Docker2Caddy - An automatic Reverse Proxy for Docker containers"
|
|
|
|
|
date: 2022-04-25
|
|
|
|
|
categories:
|
|
|
|
|
- english
|
|
|
|
|
tags:
|
|
|
|
|
- tools
|
|
|
|
|
- fsfe
|
|
|
|
|
- server
|
|
|
|
|
headerimage: /blog/docker.jpg
|
|
|
|
|
headercredits: Shipping a load of containers requires a reliable infrastructure
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
So you have a number of Docker containers running web services which you would
|
|
|
|
|
like to expose to the outside? Well, you probably will at least have considered
|
|
|
|
|
a reverse proxy already. Doing this manually for one, two or even five
|
|
|
|
|
containers may be feasible, but everything above that will be a PITA for sure.
|
|
|
|
|
At the [FSFE](https://fsfe.org) we ran into the same issue with our [own
|
|
|
|
|
distributed container
|
|
|
|
|
infrastructure](https://git.fsfe.org/fsfe-system-hackers/container-server/) at
|
|
|
|
|
and crafted a neat solution that I would like to present to you in the next few
|
|
|
|
|
minutes.
|
|
|
|
|
|
|
|
|
|
The result is
|
|
|
|
|
[Docker2Caddy](https://git.fsfe.org/fsfe-system-hackers/docker2caddy/) that
|
|
|
|
|
provides a workflow in which you can spin up new containers anytime (e.g. via a
|
|
|
|
|
CI) and the reverse proxy will just do the rest for you magically.
|
|
|
|
|
|
|
|
|
|
## The assumptions
|
|
|
|
|
|
|
|
|
|
Let's assume you want to go with reverse proxies to make your web services
|
|
|
|
|
accessible via ports 80 and 443. There are other possibilities, and in more
|
|
|
|
|
complex environments there may be already integrated solutions, but for this
|
|
|
|
|
article we'll wade in a rather simple environment spun up with `docker-compose`[^1].
|
|
|
|
|
|
|
|
|
|
Let's also assume you care about security and go with a rootless installation of
|
|
|
|
|
Docker. So the daemon will run as an unprivileged user. That's possible but much
|
|
|
|
|
more complex than the default rootful installation[^2]. Because of this, a few
|
|
|
|
|
other solutions will not work, we'll check that later.
|
|
|
|
|
|
|
|
|
|
Finally, each container shall at least have one separate domain assigned to it
|
|
|
|
|
for which you obviously want to have a valid certificate, e.g. by Let's Encrypt.
|
|
|
|
|
|
|
|
|
|
In the examples below, we have two containers running, each running a webserver
|
|
|
|
|
listening to port `8080`. The first container shall be available via
|
|
|
|
|
`first.com`, the second via `second.net`. The latter shall also be available via
|
|
|
|
|
`www.second.net`.
|
|
|
|
|
|
|
|
|
|
[^1]: [This is](https://git.fsfe.org/fsfe-system-hackers/minimal-docker) how a
|
|
|
|
|
very minimal Docker service in the FSFE infrastructure looks like. For
|
|
|
|
|
Docker2Caddy, only the `docker-compose.yml` file with its labels is
|
|
|
|
|
relevant.
|
|
|
|
|
|
|
|
|
|
[^2]: If you're interested in setting this up via Ansible, I can recommend the
|
|
|
|
|
[ansible-docker-rootless](https://github.com/konstruktoid/ansible-docker-rootless)
|
|
|
|
|
role which we integrated in our full-blown
|
|
|
|
|
[playbook](https://git.fsfe.org/fsfe-system-hackers/container-server/) for
|
|
|
|
|
the container servers.
|
|
|
|
|
|
|
|
|
|
## The problems
|
|
|
|
|
|
|
|
|
|
In the described scenario, there are a number of problem for automating the
|
|
|
|
|
configuration of the reverse proxy in order to direct a domain to the correct
|
|
|
|
|
container, starting with **container discovery** to **IPv6 routing** to
|
|
|
|
|
**handling offline containers**.
|
|
|
|
|
|
|
|
|
|
The reverse proxy has to be able to discover the currently running containers
|
|
|
|
|
and ideally monitor for changes regularly so that a newly created container with
|
|
|
|
|
a new domain is reachable within a short time without manual intervention.
|
|
|
|
|
|
|
|
|
|
Before Docker2Caddy we have used
|
|
|
|
|
[nginx-proxy](https://github.com/nginx-proxy/nginx-proxy) combined with
|
|
|
|
|
[acme-companion](https://github.com/nginx-proxy/acme-companion) (formerly known
|
|
|
|
|
as *docker-letsencrypt-nginx-proxy-companion*). These are Docker containers that
|
|
|
|
|
query all containers connected to the `bridge` Docker network. For this to work,
|
|
|
|
|
the containers have to run with environment variables indicating the desired
|
|
|
|
|
domains and local ports that shall be proxied.
|
|
|
|
|
|
|
|
|
|
In a rootless Docker setup this finally reaches its limits although discovery
|
|
|
|
|
still works. But already before that we did not like the fact that we had to
|
|
|
|
|
connect containers to the bridge network upon creation and therefore lost a bit
|
|
|
|
|
more isolation (which is dubious in Docker anyway).
|
|
|
|
|
|
|
|
|
|
Now, with rootless, IPv6 was the turning point. Even in rootful Docker setups,
|
|
|
|
|
IPv6 – a 20+ years old, well defined standard protocol – is a pain in the butt.
|
|
|
|
|
But with rootless, the FSFE System Hackers team did not manage to get IPv6
|
|
|
|
|
working in containers to the degree that we needed. While IPv6 traffic reached
|
|
|
|
|
the `nginx-proxy`, it was then treated as IPv4 traffic with the internal Docker
|
|
|
|
|
IP address. That bits you ultimately if you limit requests based on IP
|
|
|
|
|
addresses, e.g. for signups or payments. All traffic via IPv6 will be treated
|
|
|
|
|
as the same internal IPv4 address, therefore triggering the limits regularly.
|
|
|
|
|
|
|
|
|
|
The easiest solution therefore is to use a reverse proxy running on the host
|
|
|
|
|
system, not as a Docker container with its severe limitations. While the first
|
|
|
|
|
intuition lead us to nginx, we decided to go with
|
|
|
|
|
[Caddy](https://caddyserver.com/). The main advantages we saw are that a virtual
|
|
|
|
|
host in Caddy is very simple to configure and that TLS certificates are
|
|
|
|
|
generated and maintained automatically without extra dependencies like
|
|
|
|
|
`certbot`.
|
|
|
|
|
|
|
|
|
|
In this setup, containers would need to open their webserver port to the host.
|
|
|
|
|
This "public" port has to be unique per host, but the internal port can stay the
|
|
|
|
|
same, e.g. port `1234` could be mapped to port `8080` inside the container. In
|
|
|
|
|
Caddy you would then configure the domain `first.org` to forward to
|
|
|
|
|
`localhost:1234`. A more or less identical second example container could then
|
|
|
|
|
expose the port `5678` to the host, again listen on `8080` internally, and Caddy
|
|
|
|
|
would redirect `second.net` and `www.second.net` to `localhost:5678`.
|
|
|
|
|
|
|
|
|
|
But how does Caddy know about the currently running containers and the ports via
|
|
|
|
|
which they want to receive traffic? And how can we handle containers that are
|
|
|
|
|
unavailable, for instance because they crashed or have been deleted for good?
|
|
|
|
|
Docker2Caddy to the rescue!
|
|
|
|
|
|
|
|
|
|
## The solution
|
|
|
|
|
|
|
|
|
|
I already concluded that Caddy is a suitable reverse proxy for the outlined use
|
|
|
|
|
case. But in order to be care-free, the configuration has to be generated
|
|
|
|
|
automatically. For this to work, I wrote a rather simple Python application
|
|
|
|
|
called [Docker2Caddy](https://git.fsfe.org/fsfe-system-hackers/docker2caddy)
|
|
|
|
|
that is kept running in the background via a systemd service and writes proper
|
|
|
|
|
logs that are also rotated nicely.
|
|
|
|
|
|
|
|
|
|
This is how it works internally: it queries (in a configurable interval) the
|
|
|
|
|
Docker daemon for running containers. For each container it looks for specific
|
|
|
|
|
labels (that are also configurable), by default `proxy.host`, `proxy.host_alias`
|
|
|
|
|
and `proxy.port`. If one or multiple containers are found – in our case two –
|
|
|
|
|
one Caddy configuration file per container is created. This is based on a freely
|
|
|
|
|
configurable Jinja2 template. If the configuration changed, e.g. by a new host,
|
|
|
|
|
Caddy will be reloaded and will create a TLS certificate if needed.
|
|
|
|
|
|
|
|
|
|
But what happens if a container is unavailable? In Docker2Caddy you can
|
|
|
|
|
configure a grace period. Until this is reached, the Caddy configuration for the
|
|
|
|
|
container in question is not removed but could forward to a local or remote
|
|
|
|
|
error page. Only afterwards, the configuration is removed, and Caddy reloaded
|
|
|
|
|
subsequently.
|
|
|
|
|
|
|
|
|
|
So, what makes Docker2Caddy special? I am biased but see a number of points:
|
|
|
|
|
|
|
|
|
|
1. **Simplicity**: fundamentally it's a 188 pure lines of code Python script.
|
|
|
|
|
2. **Configurability**: albeit it's simplicity, it's easy to configure for
|
|
|
|
|
various needs thanks to the templates and the support for rootless Docker
|
|
|
|
|
setups.
|
|
|
|
|
3. **Adaptability**: it should be rather simple to make Docker2Caddy also work
|
|
|
|
|
for Podman, or even use different reverse proxies. Feel free to extend it
|
|
|
|
|
before I'll do it myself someday ;)
|
|
|
|
|
4. **Performance**: while I did not perform before/after benchmarks, Caddy is
|
|
|
|
|
blazingly fast and will surely perform better on the host than in a limited
|
|
|
|
|
Docker container.
|
|
|
|
|
|
|
|
|
|
If you're facing the same challenges in your setup, please feel free to try it
|
|
|
|
|
out. Installation is quite simple and there's even a [minimal Ansible
|
|
|
|
|
playbook](https://git.fsfe.org/fsfe-system-hackers/docker2caddy/src/branch/master/install.yml).
|
|
|
|
|
If you have feedback, I appreciate reading it via comments on Mastodon (see
|
|
|
|
|
below), [email](/contact), or, if you have an FSFE account, as a new issue or
|
|
|
|
|
patch at the [main repo](https://git.fsfe.org/fsfe-system-hackers/docker2caddy).
|
