add syshackers report

content/blog/2020-03-system-hackers-report-lyon.md

@@ -0,0 +1,144 @@
+---
+title: System Hackers meeting - Lyon edition
+date: 2020-03-31
+categories:
+  - english
+tags:
+  - fsfe
+  - report
+  - server
+headerimage: /blog/syshackers-group-hacking.jpg
+headercredits: Hackers in their natural working environment. For the picture we took off the black ski masks and gloves.
+
+---
+
+For the 4th time, and less than 5 months after the last meeting, the
+FSFE System Hackers met in person to coordinate their activities, work
+on complex issues, and exchange know-how. This time, we chose yet
+another home town of one of our team members as venue – Lyon in France
+where Vincent lives. What follows is a report of this gathering that
+happened shortly before *#stayhome* became the order of the day.
+
+For those who do not know this less visible but important team: The
+System Hackers are responsible for the maintenance and development of a
+[large number of services](https://wiki.fsfe.org/TechDocs/Services).
+From the fsfe.org website's deployment to the mail servers and blogs,
+from Git to internal services like DNS and monitoring, all these
+services, virtual machines and physical servers are handled by [this
+friendly group](https://wiki.fsfe.org/Teams/System-Hackers/) that is
+always looking forward to welcoming new members.
+
+Interestingly, we have gathered in the same constellation as in the
+[hackathon
+before](https://mehl.mx/blog/2019/the-3rd-fsfe-system-hackers-hackathon/),
+so Albert, Florian, Francesco, Thomas, Vincent and me tackled large and
+small challenges in the FSFE's systems. But we have also used the time
+to exchange knowledge about complex tasks and some interconnected
+systems. The official part was conducted in the fascinating [Astech
+Fablab](https://astech-fablab.fr), but word has it that
+[Ninkasi](https://www.ninkasi.fr/), an excellent pub in Lyon, was the
+actual epicentre of this year's meeting.
+
+## Sharing is caring
+
+Saturday morning after reviewing open tasks and setting our priorities,
+we started to share more knowledge about our services to reduce
+bottlenecks. For this, I drew a few diagrams to explain how we deploy
+our Docker containers, how our community database interacts with the
+mail and lists server, and how DNS works at the FSFE.
+
+To also help the non-present system hackers and "future generations",
+I've added this information to a [public wiki
+page](https://wiki.fsfe.org/TechDocs/Systems). This could also be the
+starting point to transfer more internal knowledge to public pages to
+make maintenance and onboarding easier.
+
+## Todo? Done!
+
+Afterwards, we focused on closing tasks that have been open for a longer
+time:
+
+* The DNS has been a big issue for a long time. Over the past months
+  we've migrated the source for our nameserver entries from SVN to Git,
+  rewrote our deployment scripts, and eventually upgraded the two very
+  sensitive systems to Debian 10. During the meeting, we came closer to
+  perfection: all Bind configuration cleaned from old entries, uniformly
+  formatted, and now featuring SPF, DMARC and CAA records.
+* For a better security monitoring of the 100+ mailing lists the FSFE
+  hosts, we've finalised the weekly automatic checks for sane and safe
+  settings, and a tool that helps to easily update the internal
+  documentation.
+* Speaking of monitoring: we did lack proper monitoring of our 20+ hosts
+  for availability, disk usage, TLS certificates, service status and
+  more. While we tried for a longer time to get Prometheus and Grafana
+  doing what we need, we performed a 180° turn: now, there is a Icinga2
+  installation running that already monitors a few hosts and their
+  services – [deployed with
+  Ansible](https://git.fsfe.org/fsfe-system-hackers/monitoring). In the
+  following weeks we will add more hosts and services to the watched
+  targets.
+* We plan to migrate our user-unfriendly way to share files between
+  groups to Nextcloud, including using some more of the software's
+  capabilities. During the weekend, we've tested the instance
+  thoroughly, and created some more LDAP groups that are automatically
+  transposed to groups in Nextcloud. In the same run, Albert shared some
+  more knowledge about LDAP with Vincent and me, so we get rid of more
+  bottlenecks.
+
+Then, it was time to deal with other urgent issues:
+
+* Some of us worked on making our systems more resilient against DDoS
+  attacks. Over the Christmas season, we became a target of an attack.
+  The idea is to come up with solutions that are easy to deploy on all
+  our web services while keeping complexity low. We've tested some
+  approaches and will further work on coming up with solutions.
+* Regarding webservers, we've updated the TLS configurations on various
+  services to the recommended settings, and also improved some other
+  settings while touching the configuration files.
+* We intend to ease people encrypting their emails with GnuPG. That is
+  why we experimented with WKD/WKS and will work on setting up this
+  service. As it requires some interconnection with others services,
+  this will take us some more time unfortunately.
+* On the maintenance side of things, we have upgraded all servers except
+  one to the latest Debian version, and also updated many of our Docker
+  images and containers to make use of the latest security and stability
+  improvements.
+* The FSFE hosts a few third party services, and unfortunately they have
+  been running on unmaintained systems. That is why we set up a brand
+  new host for our [sister organisation in Latin
+  America](https://fsfla.org) so they can eventually migrate, and moved
+  the [fossmarks.org](https://fossmarks.org) website to our automatic
+  CI/CD setup via Drone/Docker.
+
+
+## The next steps and developments
+
+As you can see, we completed and started to tackle a lot of issues
+again, so it won't become boring in our team any time soon. However,
+although we should know better, we intend to "change a running system"!
+
+While the in-person meetings have been highly important and also fun,
+we are in a state where knowledge and mutual trust are further
+distributed between the members, the tasks separated more clearly and
+the systems mostly well documented. So part of our feedback session was
+the question whether these meetings in the 6-12 month rhythm are still
+necessary.
+
+Yes, they are, but not more often than once a year. Instead, we would
+like to try virtual meetings and sprints. Before a sprint session, we
+would discuss all tasks (basically go through our internal Kan board),
+plan the challenges, ask for input if necessary, and resolve blockers as
+early as possible. Then, we would be prepared for a sprint day or
+afternoon during which everyone can work on their tasks while being able
+to directly contact other members. All that should happen over a video
+conference to have a more personal atmosphere.
+
+For the analogue meetings, it was requested to also plan tasks and
+priorities beforehand together, and focus on tasks that require more
+people from the group. Also, we want to have more trainings and system
+introductions like we've just had to reduce dependencies on single
+persons.
+
+All in all, this gathering has been another successful meeting and will
+set a corner stone for exciting new improvements for both the systems
+and the team. Thanks to everyone who participated!