diff --git a/content/blog/2020-03-system-hackers-report-lyon.md b/content/blog/2020-03-system-hackers-report-lyon.md new file mode 100644 index 0000000..0d93e88 --- /dev/null +++ b/content/blog/2020-03-system-hackers-report-lyon.md @@ -0,0 +1,144 @@ +--- +title: System Hackers meeting - Lyon edition +date: 2020-03-31 +categories: + - english +tags: + - fsfe + - report + - server +headerimage: /blog/syshackers-group-hacking.jpg +headercredits: Hackers in their natural working environment. For the picture we took off the black ski masks and gloves. + +--- + +For the 4th time, and less than 5 months after the last meeting, the +FSFE System Hackers met in person to coordinate their activities, work +on complex issues, and exchange know-how. This time, we chose yet +another home town of one of our team members as venue – Lyon in France +where Vincent lives. What follows is a report of this gathering that +happened shortly before *#stayhome* became the order of the day. + +For those who do not know this less visible but important team: The +System Hackers are responsible for the maintenance and development of a +[large number of services](https://wiki.fsfe.org/TechDocs/Services). +From the fsfe.org website's deployment to the mail servers and blogs, +from Git to internal services like DNS and monitoring, all these +services, virtual machines and physical servers are handled by [this +friendly group](https://wiki.fsfe.org/Teams/System-Hackers/) that is +always looking forward to welcoming new members. + +Interestingly, we have gathered in the same constellation as in the +[hackathon +before](https://mehl.mx/blog/2019/the-3rd-fsfe-system-hackers-hackathon/), +so Albert, Florian, Francesco, Thomas, Vincent and me tackled large and +small challenges in the FSFE's systems. But we have also used the time +to exchange knowledge about complex tasks and some interconnected +systems. The official part was conducted in the fascinating [Astech +Fablab](https://astech-fablab.fr), but word has it that +[Ninkasi](https://www.ninkasi.fr/), an excellent pub in Lyon, was the +actual epicentre of this year's meeting. + +## Sharing is caring + +Saturday morning after reviewing open tasks and setting our priorities, +we started to share more knowledge about our services to reduce +bottlenecks. For this, I drew a few diagrams to explain how we deploy +our Docker containers, how our community database interacts with the +mail and lists server, and how DNS works at the FSFE. + +To also help the non-present system hackers and "future generations", +I've added this information to a [public wiki +page](https://wiki.fsfe.org/TechDocs/Systems). This could also be the +starting point to transfer more internal knowledge to public pages to +make maintenance and onboarding easier. + +## Todo? Done! + +Afterwards, we focused on closing tasks that have been open for a longer +time: + +* The DNS has been a big issue for a long time. Over the past months + we've migrated the source for our nameserver entries from SVN to Git, + rewrote our deployment scripts, and eventually upgraded the two very + sensitive systems to Debian 10. During the meeting, we came closer to + perfection: all Bind configuration cleaned from old entries, uniformly + formatted, and now featuring SPF, DMARC and CAA records. +* For a better security monitoring of the 100+ mailing lists the FSFE + hosts, we've finalised the weekly automatic checks for sane and safe + settings, and a tool that helps to easily update the internal + documentation. +* Speaking of monitoring: we did lack proper monitoring of our 20+ hosts + for availability, disk usage, TLS certificates, service status and + more. While we tried for a longer time to get Prometheus and Grafana + doing what we need, we performed a 180° turn: now, there is a Icinga2 + installation running that already monitors a few hosts and their + services – [deployed with + Ansible](https://git.fsfe.org/fsfe-system-hackers/monitoring). In the + following weeks we will add more hosts and services to the watched + targets. +* We plan to migrate our user-unfriendly way to share files between + groups to Nextcloud, including using some more of the software's + capabilities. During the weekend, we've tested the instance + thoroughly, and created some more LDAP groups that are automatically + transposed to groups in Nextcloud. In the same run, Albert shared some + more knowledge about LDAP with Vincent and me, so we get rid of more + bottlenecks. + +Then, it was time to deal with other urgent issues: + +* Some of us worked on making our systems more resilient against DDoS + attacks. Over the Christmas season, we became a target of an attack. + The idea is to come up with solutions that are easy to deploy on all + our web services while keeping complexity low. We've tested some + approaches and will further work on coming up with solutions. +* Regarding webservers, we've updated the TLS configurations on various + services to the recommended settings, and also improved some other + settings while touching the configuration files. +* We intend to ease people encrypting their emails with GnuPG. That is + why we experimented with WKD/WKS and will work on setting up this + service. As it requires some interconnection with others services, + this will take us some more time unfortunately. +* On the maintenance side of things, we have upgraded all servers except + one to the latest Debian version, and also updated many of our Docker + images and containers to make use of the latest security and stability + improvements. +* The FSFE hosts a few third party services, and unfortunately they have + been running on unmaintained systems. That is why we set up a brand + new host for our [sister organisation in Latin + America](https://fsfla.org) so they can eventually migrate, and moved + the [fossmarks.org](https://fossmarks.org) website to our automatic + CI/CD setup via Drone/Docker. + + +## The next steps and developments + +As you can see, we completed and started to tackle a lot of issues +again, so it won't become boring in our team any time soon. However, +although we should know better, we intend to "change a running system"! + +While the in-person meetings have been highly important and also fun, +we are in a state where knowledge and mutual trust are further +distributed between the members, the tasks separated more clearly and +the systems mostly well documented. So part of our feedback session was +the question whether these meetings in the 6-12 month rhythm are still +necessary. + +Yes, they are, but not more often than once a year. Instead, we would +like to try virtual meetings and sprints. Before a sprint session, we +would discuss all tasks (basically go through our internal Kan board), +plan the challenges, ask for input if necessary, and resolve blockers as +early as possible. Then, we would be prepared for a sprint day or +afternoon during which everyone can work on their tasks while being able +to directly contact other members. All that should happen over a video +conference to have a more personal atmosphere. + +For the analogue meetings, it was requested to also plan tasks and +priorities beforehand together, and focus on tasks that require more +people from the group. Also, we want to have more trainings and system +introductions like we've just had to reduce dependencies on single +persons. + +All in all, this gathering has been another successful meeting and will +set a corner stone for exciting new improvements for both the systems +and the team. Thanks to everyone who participated! diff --git a/static/img/blog/syshackers-group-hacking.jpg b/static/img/blog/syshackers-group-hacking.jpg new file mode 100644 index 0000000..d96153b Binary files /dev/null and b/static/img/blog/syshackers-group-hacking.jpg differ