remove hardlinebreak because it creates issues with normal line-breaks

config.toml

@@ -7,7 +7,7 @@ pygmentsCodeFences = true
 
 [blackfriday]
 # preserves linebreaks, and transforms to <br />
-extensions = [ "hardLineBreak" ]
+extensions = [ "" ]
 
 [permalinks]
   blog = "/blog/:year/:slug"

content/blog/2019-10-system-hackers-meeting-report.md

@@ -0,0 +1,130 @@
+---
+title: Tales from the 3rd FSFE System Hackers hackathon
+date: 2019-10-22
+categories:
+  - english
+tags:
+  - fsfe
+  - report
+  - server
+draft: true
+headerimage: /blog/system-chaos.jpg
+headercredits: Fortunately not how the FSFE's infrastructure looks like
+
+---
+
+On 10 and 11 October, the FSFE System Hackers met in person to tackle
+problems and new features regarding the servers and services the FSFE
+is running. The team consists of dedicated volunteers who ensure that
+the community and staff can work effectively. The recent meeting built
+on the great work of the past 2 years which have been shaped by large
+personal and technical changes.
+
+The System Hackers are responsible for the maintenance and development
+of a [large number of
+services](https://wiki.fsfe.org/TechDocs/Services). From the fsfe.org
+website's deployment to the mail servers and blogs, from Git to
+internal services like DNS and monitoring, all these services, virtual
+machines and physical servers are handled by [this friendly
+group](https://wiki.fsfe.org/Teams/System-Hackers/) that is always
+looking forward to welcoming new members.
+
+{{< figure src="/img/blog/system-servers.png" caption="Overview of the FSFE's services and servers" >}}
+
+So in October, six of us met in Cologne. Fittingly, according to a
+saying in this region, if you do something for the third time, it's
+already tradition. So we accomplished this after successful meetings in
+Berlin (April 2018) and Vienna (March 2019). And although it took place
+on workdays, it's been the meeting with the highest participation so
+far!
+
+## Getting. Things. Done!
+
+After the first and second meeting were mostly about getting an
+overview of historically grown and sparsely documented infrastructure
+and bringing it into a stable state, we were able to deal with a few
+more general topics this time. At the same time, we exchanged our
+knowledge with newly joined team members. Please find the areas we
+worked on below:
+
+* Florian migrated the FSFE Blogs to a new server and thereby also
+  updated the underlying Wordpress to the latest version. This has been
+  a major blocker for several other tasks and our largest security risk.
+  There are still a few things left to do, e.g. creating a theme in line
+  with the FSFE design and some announcement to the community. However,
+  the most complicated part is done!
+* Altogether, we upgraded a lot of machines to Debian 10, just after we lifted most
+  servers to Debian 9 in March. Some are still missing, but since the
+  migration is rather painless, we can do that during the next months.
+* We confirmed that the new decentralised backup system setup by myself
+  and based on Borg works fine. This gives us more confidence in our
+  infrastructure.
+* Thanks to Florian and Albert, we finally got rid of the last 2
+  services that were not using Let's Encrypt's self-renewing
+  certificates.
+* Vincent and Francesco took care of finishing the migration of all our Docker containers
+  to use the Docker-in-Docker deployment instead of the hacky Ansible
+  playbooks we used initially. This has a few security advantages and
+  enables the next developments for a more resilient Docker
+  infrastructure.
+* At the moment, all our Docker containers run on one single virtual
+  machine. Although this runs on a Proxmox/Ceph cluster, it's obviously
+  a single point of failure. However, for a distribution on multiple
+  servers we lack the hardware resources. Nonetheless, we already have
+  concrete plans how to make the Docker setup more resilient as soon as
+  we have more hardware available. Vincent documented this on [a wiki
+  page](https://wiki.fsfe.org/TechDocs/Docker/docker-machine).
+* On the human side, we made sure that all of us know what's on the
+  plate for the next weeks and months. We have quite a few open issues
+  collected in our Kanban board, and we quickly went through all of them
+  to sketch the possible next steps and distribute responsibilities.
+
+
+## Started projects in the making
+
+Two days are quite some time and we worked hard to use them as
+effectively as possible, so some tasks have been started but could not
+be completed – partly because we just did no have enough time, partly
+because they require more coordination and in-depth discussion:
+
+* As follow-up on a few unpleasant surprises with Mailman's default
+  values, we figured that it is important to have an automatic overview
+  of the most sensible settings of the 127 (!) mailing lists we host.
+  Vincent started to work on a way to extract this information in a
+  human- and machine-readable format and merge/compare it with the more
+  verbose documentation on the mailing lists we have internally.
+* Francesco tackled a different weak point we have: monitoring. We lack
+  a tool that informs us immediately about problems in our
+  infrastructure, e.g. defunct core services, full disk drives or
+  expired certificates. Since this is not trivial at all, it requires
+  some more time.
+* Thomas, maintainer of the FSFE wiki, researched on a way to better organise and distribute the SSH
+  accesses in our team. Right now, we have no comfortable way to add or
+  remove SSH keys on our more than 20 machines. His idea is to use an
+  Ansible playbook to manage these, and thereby also create a shared
+  Ansible inventory which can be used as a submodule for the other
+  playbooks we use in the team so we don't have to maintain all of them
+  individually if a machine is added, changed or removed.
+* One of the most ancient physical machines we still run is hosting the
+  SVN service which is only used by one service now: DNS. We started to work
+  on migrating that over to Git and simultaneously improving the
+  error-checking of the DNS configuration. Albert and I will continue
+  with that gradually.
+* Not on the system hackers meeting itself but two days later, Björn,
+  Albert and I worked on getting a Nextcloud instance running. Caused by
+  our rather special LDAP setup, we had to debug a lot of strange
+  behaviour but finally figured everything out. Now, the last missing
+  blocker is some user/permission setting within our LDAP. As soon as
+  this is finished, we can shut down one more historically grown,
+  customised-hacked and user-unfriendly service.
+
+
+Overall, the perspective for the System Hackers is better than ever. We
+are a growing team carried by motivated and skilled volunteers with a
+shared vision of how the systems should develop. At the same time, we
+have a lot of public and internal documentation available to make it
+easy for new people to join us.
+
+I would like to thank Albert, Florian, Francesco, Thomas and Vincent for
+their participation in this meeting, and them and all other System
+Hackers for their dedication to keep the FSFE running!

content/contact.md

@@ -3,15 +3,15 @@ title: Contact
 page: true
 ---
 
-Email FSFE: {{< cloakemail address="max.mehl@fsfe.org" >}}
-Email private: {{< cloakemail address="mail@mehl.mx" >}}
-XMPP: {{< cloakemail address="max.mehl@jabber.fsfe.org" protocol="xmpp" >}}
+Email FSFE: {{< cloakemail address="max.mehl@fsfe.org" >}}<br />
+Email private: {{< cloakemail address="mail@mehl.mx" >}}<br />
+XMPP: {{< cloakemail address="max.mehl@jabber.fsfe.org" protocol="xmpp" >}}<br />
 
-Mastodon: [@mxmehl@mastodon.social](https://mastodon.social/@mxmehl)
-Twitter: [@mxmehl](https://twitter.com/mxmehl)
-Weblog: [mehl.mx/blog](/blog)
-Diaspora: [mxmehl@diasp.eu](https://diasp.eu/u/mxmehl)
-LinkedIn: [mxmehl ](https://linkedin.com/in/mxmehl)
+Mastodon: [@mxmehl@mastodon.social](https://mastodon.social/@mxmehl)<br />
+Twitter: [@mxmehl](https://twitter.com/mxmehl)<br />
+Weblog: [mehl.mx/blog](/blog)<br />
+Diaspora: [mxmehl@diasp.eu](https://diasp.eu/u/mxmehl)<br />
+LinkedIn: [mxmehl ](https://linkedin.com/in/mxmehl)<br />
 
-GnuPG: [0x371E2E92](https://up.mehl.mx/mehl-current.pub.asc)
+GnuPG: [0x371E2E92](https://up.mehl.mx/mehl-current.pub.asc)<br />
 OMEMO: [OMEMO Fingerprints](https://up.mehl.mx/mehl-omemo.asc)