---
title: "Docker2Caddy - An automatic Reverse Proxy for Docker containers"
date: 2022-04-25
categories:
  - blog
  - english
tags:
  - FSFE
  - SystemAdministration
  - Code
headerimage:
   src: /blog/docker.jpg
   text: Shipping a load of containers requires a reliable infrastructure
mastodon_toot_url: "https://mastodon.social/@mxmehl/108193478971435955"
---

So you have a number of Docker containers running web services which you would
like to expose to the outside? Well, you probably will at least have considered
a reverse proxy already. Doing this manually for one, two or even five
containers may be feasible, but everything above that will be a PITA for sure.
At the [FSFE](https://fsfe.org) we ran into the same issue with our [own
distributed container
infrastructure](https://git.fsfe.org/fsfe-system-hackers/container-server/) at
and crafted a neat solution that I would like to present to you in the next few
minutes.

The result is
[Docker2Caddy](https://git.fsfe.org/fsfe-system-hackers/docker2caddy/) that
provides a workflow in which you can spin up new containers anytime (e.g. via a
CI) and the reverse proxy will just do the rest for you magically.

## The assumptions

Let's assume you want to go with reverse proxies to make your web services
accessible via ports 80 and 443. There are other possibilities, and in more
complex environments there may be already integrated solutions, but for this
article we'll wade in a rather simple environment spun up with `docker-compose`[^1].

Let's also assume you care about security and go with a rootless installation of
Docker. So the daemon will run as an unprivileged user. That's possible but much
more complex than the default rootful installation[^2]. Because of this, a few
other solutions will not work, we'll check that later.

Finally, each container shall at least have one separate domain assigned to it
for which you obviously want to have a valid certificate, e.g. by Let's Encrypt.

In the examples below, we have two containers running, each running a webserver
listening to port `8080`. The first container shall be available via
`first.com`, the second via `second.net`. The latter shall also be available via
`www.second.net`.

[^1]: [This is](https://git.fsfe.org/fsfe-system-hackers/minimal-docker) how a
    very minimal Docker service in the FSFE infrastructure looks like. For
    Docker2Caddy, only the `docker-compose.yml` file with its labels is
    relevant.

[^2]: If you're interested in setting this up via Ansible, I can recommend the
    [ansible-docker-rootless](https://github.com/konstruktoid/ansible-docker-rootless)
    role which we integrated in our full-blown
    [playbook](https://git.fsfe.org/fsfe-system-hackers/container-server/) for
    the container servers.

## The problems

In the described scenario, there are a number of problem for automating the
configuration of the reverse proxy in order to direct a domain to the correct
container, starting with **container discovery** to **IPv6 routing** to
**handling offline containers**.

The reverse proxy has to be able to discover the currently running containers
and ideally monitor for changes regularly so that a newly created container with
a new domain is reachable within a short time without manual intervention.

Before Docker2Caddy we have used
[nginx-proxy](https://github.com/nginx-proxy/nginx-proxy) combined with
[acme-companion](https://github.com/nginx-proxy/acme-companion) (formerly known
as *docker-letsencrypt-nginx-proxy-companion*). These are Docker containers that
query all containers connected to the `bridge` Docker network. For this to work,
the containers have to run with environment variables indicating the desired
domains and local ports that shall be proxied.

In a rootless Docker setup this finally reaches its limits although discovery
still works. But already before that we did not like the fact that we had to
connect containers to the bridge network upon creation and therefore lost a bit
more isolation (which is dubious in Docker anyway).

Now, with rootless, IPv6 was the turning point. Even in rootful Docker setups,
IPv6 – a 20+ years old, well defined standard protocol – is a pain in the butt.
But with rootless, the FSFE System Hackers team did not manage to get IPv6
working in containers to the degree that we needed. While IPv6 traffic reached
the `nginx-proxy`, it was then treated as IPv4 traffic with the internal Docker
IP address. That bits you ultimately if you limit requests based on IP
addresses, e.g. for signups or payments. All traffic via IPv6 will be treated
as the same internal IPv4 address, therefore triggering the limits regularly.

The easiest solution therefore is to use a reverse proxy running on the host
system, not as a Docker container with its severe limitations. While the first
intuition lead us to nginx, we decided to go with
[Caddy](https://caddyserver.com/). The main advantages we saw are that a virtual
host in Caddy is very simple to configure and that TLS certificates are
generated and maintained automatically without extra dependencies like
`certbot`.

In this setup, containers would need to open their webserver port to the host.
This "public" port has to be unique per host, but the internal port can stay the
same, e.g. port `1234` could be mapped to port `8080` inside the container. In
Caddy you would then configure the domain `first.org` to forward to
`localhost:1234`. A more or less identical second example container could then
expose the port `5678` to the host, again listen on `8080` internally, and Caddy
would redirect `second.net` and `www.second.net` to `localhost:5678`.

But how does Caddy know about the currently running containers and the ports via
which they want to receive traffic? And how can we handle containers that are
unavailable, for instance because they crashed or have been deleted for good?
Docker2Caddy to the rescue!

## The solution

I already concluded that Caddy is a suitable reverse proxy for the outlined use
case. But in order to be care-free, the configuration has to be generated
automatically. For this to work, I wrote a rather simple Python application
called [Docker2Caddy](https://git.fsfe.org/fsfe-system-hackers/docker2caddy)
that is kept running in the background via a systemd service and writes proper
logs that are also rotated nicely.

This is how it works internally: it queries (in a configurable interval) the
Docker daemon for running containers. For each container it looks for specific
labels (that are also configurable), by default `proxy.host`, `proxy.host_alias`
and `proxy.port`. If one or multiple containers are found – in our case two –
one Caddy configuration file per container is created. This is based on a freely
configurable Jinja2 template. If the configuration changed, e.g. by a new host,
Caddy will be reloaded and will create a TLS certificate if needed.

But what happens if a container is unavailable? In Docker2Caddy you can
configure a grace period. Until this is reached, the Caddy configuration for the
container in question is not removed but could forward to a local or remote
error page. Only afterwards, the configuration is removed, and Caddy reloaded
subsequently.

So, what makes Docker2Caddy special? I am biased but see a number of points:

1. **Simplicity**: fundamentally it's a 188 pure lines of code Python script.
2. **Configurability**: albeit it's simplicity, it's easy to configure for
   various needs thanks to the templates and the support for rootless Docker
   setups.
3. **Adaptability**: it should be rather simple to make Docker2Caddy also work
   for Podman, or even use different reverse proxies. Feel free to extend it
   before I'll do it myself someday ;)
4. **Performance**: while I did not perform before/after benchmarks, Caddy is
   blazingly fast and will surely perform better on the host than in a limited
   Docker container.

If you're facing the same challenges in your setup, please feel free to try it
out. Installation is quite simple and there's even a [minimal Ansible
playbook](https://git.fsfe.org/fsfe-system-hackers/docker2caddy/src/branch/master/install.yml).
If you have feedback, I appreciate reading it via comments on Mastodon (see
below), [email](/contact), or, if you have an FSFE account, as a new issue or
patch at the [main repo](https://git.fsfe.org/fsfe-system-hackers/docker2caddy).
