Browse Source

add syshackers report

max.mehl 2 months ago
parent
commit
05f03f2a96
No account linked to committer's email address

+ 144
- 0
content/blog/2020-03-system-hackers-report-lyon.md View File

@@ -0,0 +1,144 @@
1
+---
2
+title: System Hackers meeting - Lyon edition
3
+date: 2020-03-31
4
+categories:
5
+  - english
6
+tags:
7
+  - fsfe
8
+  - report
9
+  - server
10
+headerimage: /blog/syshackers-group-hacking.jpg
11
+headercredits: Hackers in their natural working environment. For the picture we took off the black ski masks and gloves.
12
+
13
+---
14
+
15
+For the 4th time, and less than 5 months after the last meeting, the
16
+FSFE System Hackers met in person to coordinate their activities, work
17
+on complex issues, and exchange know-how. This time, we chose yet
18
+another home town of one of our team members as venue – Lyon in France
19
+where Vincent lives. What follows is a report of this gathering that
20
+happened shortly before *#stayhome* became the order of the day.
21
+
22
+For those who do not know this less visible but important team: The
23
+System Hackers are responsible for the maintenance and development of a
24
+[large number of services](https://wiki.fsfe.org/TechDocs/Services).
25
+From the fsfe.org website's deployment to the mail servers and blogs,
26
+from Git to internal services like DNS and monitoring, all these
27
+services, virtual machines and physical servers are handled by [this
28
+friendly group](https://wiki.fsfe.org/Teams/System-Hackers/) that is
29
+always looking forward to welcoming new members.
30
+
31
+Interestingly, we have gathered in the same constellation as in the
32
+[hackathon
33
+before](https://mehl.mx/blog/2019/the-3rd-fsfe-system-hackers-hackathon/),
34
+so Albert, Florian, Francesco, Thomas, Vincent and me tackled large and
35
+small challenges in the FSFE's systems. But we have also used the time
36
+to exchange knowledge about complex tasks and some interconnected
37
+systems. The official part was conducted in the fascinating [Astech
38
+Fablab](https://astech-fablab.fr), but word has it that
39
+[Ninkasi](https://www.ninkasi.fr/), an excellent pub in Lyon, was the
40
+actual epicentre of this year's meeting.
41
+
42
+## Sharing is caring
43
+
44
+Saturday morning after reviewing open tasks and setting our priorities,
45
+we started to share more knowledge about our services to reduce
46
+bottlenecks. For this, I drew a few diagrams to explain how we deploy
47
+our Docker containers, how our community database interacts with the
48
+mail and lists server, and how DNS works at the FSFE.
49
+
50
+To also help the non-present system hackers and "future generations",
51
+I've added this information to a [public wiki
52
+page](https://wiki.fsfe.org/TechDocs/Systems). This could also be the
53
+starting point to transfer more internal knowledge to public pages to
54
+make maintenance and onboarding easier.
55
+
56
+## Todo? Done!
57
+
58
+Afterwards, we focused on closing tasks that have been open for a longer
59
+time:
60
+
61
+* The DNS has been a big issue for a long time. Over the past months
62
+  we've migrated the source for our nameserver entries from SVN to Git,
63
+  rewrote our deployment scripts, and eventually upgraded the two very
64
+  sensitive systems to Debian 10. During the meeting, we came closer to
65
+  perfection: all Bind configuration cleaned from old entries, uniformly
66
+  formatted, and now featuring SPF, DMARC and CAA records.
67
+* For a better security monitoring of the 100+ mailing lists the FSFE
68
+  hosts, we've finalised the weekly automatic checks for sane and safe
69
+  settings, and a tool that helps to easily update the internal
70
+  documentation.
71
+* Speaking of monitoring: we did lack proper monitoring of our 20+ hosts
72
+  for availability, disk usage, TLS certificates, service status and
73
+  more. While we tried for a longer time to get Prometheus and Grafana
74
+  doing what we need, we performed a 180° turn: now, there is a Icinga2
75
+  installation running that already monitors a few hosts and their
76
+  services – [deployed with
77
+  Ansible](https://git.fsfe.org/fsfe-system-hackers/monitoring). In the
78
+  following weeks we will add more hosts and services to the watched
79
+  targets.
80
+* We plan to migrate our user-unfriendly way to share files between
81
+  groups to Nextcloud, including using some more of the software's
82
+  capabilities. During the weekend, we've tested the instance
83
+  thoroughly, and created some more LDAP groups that are automatically
84
+  transposed to groups in Nextcloud. In the same run, Albert shared some
85
+  more knowledge about LDAP with Vincent and me, so we get rid of more
86
+  bottlenecks.
87
+
88
+Then, it was time to deal with other urgent issues:
89
+
90
+* Some of us worked on making our systems more resilient against DDoS
91
+  attacks. Over the Christmas season, we became a target of an attack.
92
+  The idea is to come up with solutions that are easy to deploy on all
93
+  our web services while keeping complexity low. We've tested some
94
+  approaches and will further work on coming up with solutions.
95
+* Regarding webservers, we've updated the TLS configurations on various
96
+  services to the recommended settings, and also improved some other
97
+  settings while touching the configuration files.
98
+* We intend to ease people encrypting their emails with GnuPG. That is
99
+  why we experimented with WKD/WKS and will work on setting up this
100
+  service. As it requires some interconnection with others services,
101
+  this will take us some more time unfortunately.
102
+* On the maintenance side of things, we have upgraded all servers except
103
+  one to the latest Debian version, and also updated many of our Docker
104
+  images and containers to make use of the latest security and stability
105
+  improvements.
106
+* The FSFE hosts a few third party services, and unfortunately they have
107
+  been running on unmaintained systems. That is why we set up a brand
108
+  new host for our [sister organisation in Latin
109
+  America](https://fsfla.org) so they can eventually migrate, and moved
110
+  the [fossmarks.org](https://fossmarks.org) website to our automatic
111
+  CI/CD setup via Drone/Docker.
112
+
113
+
114
+## The next steps and developments
115
+
116
+As you can see, we completed and started to tackle a lot of issues
117
+again, so it won't become boring in our team any time soon. However,
118
+although we should know better, we intend to "change a running system"!
119
+
120
+While the in-person meetings have been highly important and also fun,
121
+we are in a state where knowledge and mutual trust are further
122
+distributed between the members, the tasks separated more clearly and
123
+the systems mostly well documented. So part of our feedback session was
124
+the question whether these meetings in the 6-12 month rhythm are still
125
+necessary.
126
+
127
+Yes, they are, but not more often than once a year. Instead, we would
128
+like to try virtual meetings and sprints. Before a sprint session, we
129
+would discuss all tasks (basically go through our internal Kan board),
130
+plan the challenges, ask for input if necessary, and resolve blockers as
131
+early as possible. Then, we would be prepared for a sprint day or
132
+afternoon during which everyone can work on their tasks while being able
133
+to directly contact other members. All that should happen over a video
134
+conference to have a more personal atmosphere.
135
+
136
+For the analogue meetings, it was requested to also plan tasks and
137
+priorities beforehand together, and focus on tasks that require more
138
+people from the group. Also, we want to have more trainings and system
139
+introductions like we've just had to reduce dependencies on single
140
+persons.
141
+
142
+All in all, this gathering has been another successful meeting and will
143
+set a corner stone for exciting new improvements for both the systems
144
+and the team. Thanks to everyone who participated!

BIN
static/img/blog/syshackers-group-hacking.jpg View File


Loading…
Cancel
Save