Browse Source

remove hardlinebreak because it creates issues with normal line-breaks

max.mehl 3 weeks ago
parent
commit
4c5ad0637d
No account linked to committer's email address

+ 1
- 1
config.toml View File

@@ -7,7 +7,7 @@ pygmentsCodeFences = true
7 7
 
8 8
 [blackfriday]
9 9
 # preserves linebreaks, and transforms to <br />
10
-extensions = [ "hardLineBreak" ]
10
+extensions = [ "" ]
11 11
 
12 12
 [permalinks]
13 13
   blog = "/blog/:year/:slug"

+ 130
- 0
content/blog/2019-10-system-hackers-meeting-report.md View File

@@ -0,0 +1,130 @@
1
+---
2
+title: Tales from the 3rd FSFE System Hackers hackathon
3
+date: 2019-10-22
4
+categories:
5
+  - english
6
+tags:
7
+  - fsfe
8
+  - report
9
+  - server
10
+draft: true
11
+headerimage: /blog/system-chaos.jpg
12
+headercredits: Fortunately not how the FSFE's infrastructure looks like
13
+
14
+---
15
+
16
+On 10 and 11 October, the FSFE System Hackers met in person to tackle
17
+problems and new features regarding the servers and services the FSFE
18
+is running. The team consists of dedicated volunteers who ensure that
19
+the community and staff can work effectively. The recent meeting built
20
+on the great work of the past 2 years which have been shaped by large
21
+personal and technical changes.
22
+
23
+The System Hackers are responsible for the maintenance and development
24
+of a [large number of
25
+services](https://wiki.fsfe.org/TechDocs/Services). From the fsfe.org
26
+website's deployment to the mail servers and blogs, from Git to
27
+internal services like DNS and monitoring, all these services, virtual
28
+machines and physical servers are handled by [this friendly
29
+group](https://wiki.fsfe.org/Teams/System-Hackers/) that is always
30
+looking forward to welcoming new members.
31
+
32
+{{< figure src="/img/blog/system-servers.png" caption="Overview of the FSFE's services and servers" >}}
33
+
34
+So in October, six of us met in Cologne. Fittingly, according to a
35
+saying in this region, if you do something for the third time, it's
36
+already tradition. So we accomplished this after successful meetings in
37
+Berlin (April 2018) and Vienna (March 2019). And although it took place
38
+on workdays, it's been the meeting with the highest participation so
39
+far!
40
+
41
+## Getting. Things. Done!
42
+
43
+After the first and second meeting were mostly about getting an
44
+overview of historically grown and sparsely documented infrastructure
45
+and bringing it into a stable state, we were able to deal with a few
46
+more general topics this time. At the same time, we exchanged our
47
+knowledge with newly joined team members. Please find the areas we
48
+worked on below:
49
+
50
+* Florian migrated the FSFE Blogs to a new server and thereby also
51
+  updated the underlying Wordpress to the latest version. This has been
52
+  a major blocker for several other tasks and our largest security risk.
53
+  There are still a few things left to do, e.g. creating a theme in line
54
+  with the FSFE design and some announcement to the community. However,
55
+  the most complicated part is done!
56
+* Altogether, we upgraded a lot of machines to Debian 10, just after we lifted most
57
+  servers to Debian 9 in March. Some are still missing, but since the
58
+  migration is rather painless, we can do that during the next months.
59
+* We confirmed that the new decentralised backup system setup by myself
60
+  and based on Borg works fine. This gives us more confidence in our
61
+  infrastructure.
62
+* Thanks to Florian and Albert, we finally got rid of the last 2
63
+  services that were not using Let's Encrypt's self-renewing
64
+  certificates.
65
+* Vincent and Francesco took care of finishing the migration of all our Docker containers
66
+  to use the Docker-in-Docker deployment instead of the hacky Ansible
67
+  playbooks we used initially. This has a few security advantages and
68
+  enables the next developments for a more resilient Docker
69
+  infrastructure.
70
+* At the moment, all our Docker containers run on one single virtual
71
+  machine. Although this runs on a Proxmox/Ceph cluster, it's obviously
72
+  a single point of failure. However, for a distribution on multiple
73
+  servers we lack the hardware resources. Nonetheless, we already have
74
+  concrete plans how to make the Docker setup more resilient as soon as
75
+  we have more hardware available. Vincent documented this on [a wiki
76
+  page](https://wiki.fsfe.org/TechDocs/Docker/docker-machine).
77
+* On the human side, we made sure that all of us know what's on the
78
+  plate for the next weeks and months. We have quite a few open issues
79
+  collected in our Kanban board, and we quickly went through all of them
80
+  to sketch the possible next steps and distribute responsibilities.
81
+
82
+
83
+## Started projects in the making
84
+
85
+Two days are quite some time and we worked hard to use them as
86
+effectively as possible, so some tasks have been started but could not
87
+be completed – partly because we just did no have enough time, partly
88
+because they require more coordination and in-depth discussion:
89
+
90
+* As follow-up on a few unpleasant surprises with Mailman's default
91
+  values, we figured that it is important to have an automatic overview
92
+  of the most sensible settings of the 127 (!) mailing lists we host.
93
+  Vincent started to work on a way to extract this information in a
94
+  human- and machine-readable format and merge/compare it with the more
95
+  verbose documentation on the mailing lists we have internally.
96
+* Francesco tackled a different weak point we have: monitoring. We lack
97
+  a tool that informs us immediately about problems in our
98
+  infrastructure, e.g. defunct core services, full disk drives or
99
+  expired certificates. Since this is not trivial at all, it requires
100
+  some more time.
101
+* Thomas, maintainer of the FSFE wiki, researched on a way to better organise and distribute the SSH
102
+  accesses in our team. Right now, we have no comfortable way to add or
103
+  remove SSH keys on our more than 20 machines. His idea is to use an
104
+  Ansible playbook to manage these, and thereby also create a shared
105
+  Ansible inventory which can be used as a submodule for the other
106
+  playbooks we use in the team so we don't have to maintain all of them
107
+  individually if a machine is added, changed or removed.
108
+* One of the most ancient physical machines we still run is hosting the
109
+  SVN service which is only used by one service now: DNS. We started to work
110
+  on migrating that over to Git and simultaneously improving the
111
+  error-checking of the DNS configuration. Albert and I will continue
112
+  with that gradually.
113
+* Not on the system hackers meeting itself but two days later, Björn,
114
+  Albert and I worked on getting a Nextcloud instance running. Caused by
115
+  our rather special LDAP setup, we had to debug a lot of strange
116
+  behaviour but finally figured everything out. Now, the last missing
117
+  blocker is some user/permission setting within our LDAP. As soon as
118
+  this is finished, we can shut down one more historically grown,
119
+  customised-hacked and user-unfriendly service.
120
+
121
+
122
+Overall, the perspective for the System Hackers is better than ever. We
123
+are a growing team carried by motivated and skilled volunteers with a
124
+shared vision of how the systems should develop. At the same time, we
125
+have a lot of public and internal documentation available to make it
126
+easy for new people to join us.
127
+
128
+I would like to thank Albert, Florian, Francesco, Thomas and Vincent for
129
+their participation in this meeting, and them and all other System
130
+Hackers for their dedication to keep the FSFE running!

+ 9
- 9
content/contact.md View File

@@ -3,15 +3,15 @@ title: Contact
3 3
 page: true
4 4
 ---
5 5
 
6
-Email FSFE: {{< cloakemail address="max.mehl@fsfe.org" >}}
7
-Email private: {{< cloakemail address="mail@mehl.mx" >}}
8
-XMPP: {{< cloakemail address="max.mehl@jabber.fsfe.org" protocol="xmpp" >}}
6
+Email FSFE: {{< cloakemail address="max.mehl@fsfe.org" >}}<br />
7
+Email private: {{< cloakemail address="mail@mehl.mx" >}}<br />
8
+XMPP: {{< cloakemail address="max.mehl@jabber.fsfe.org" protocol="xmpp" >}}<br />
9 9
 
10
-Mastodon: [@mxmehl@mastodon.social](https://mastodon.social/@mxmehl)
11
-Twitter: [@mxmehl](https://twitter.com/mxmehl)
12
-Weblog: [mehl.mx/blog](/blog)
13
-Diaspora: [mxmehl@diasp.eu](https://diasp.eu/u/mxmehl)
14
-LinkedIn: [mxmehl ](https://linkedin.com/in/mxmehl)
10
+Mastodon: [@mxmehl@mastodon.social](https://mastodon.social/@mxmehl)<br />
11
+Twitter: [@mxmehl](https://twitter.com/mxmehl)<br />
12
+Weblog: [mehl.mx/blog](/blog)<br />
13
+Diaspora: [mxmehl@diasp.eu](https://diasp.eu/u/mxmehl)<br />
14
+LinkedIn: [mxmehl ](https://linkedin.com/in/mxmehl)<br />
15 15
 
16
-GnuPG: [0x371E2E92](https://up.mehl.mx/mehl-current.pub.asc)
16
+GnuPG: [0x371E2E92](https://up.mehl.mx/mehl-current.pub.asc)<br />
17 17
 OMEMO: [OMEMO Fingerprints](https://up.mehl.mx/mehl-omemo.asc)

BIN
static/img/blog/system-chaos.jpg View File


BIN
static/img/blog/system-servers.png View File


Loading…
Cancel
Save