1 00:00:00,000 --> 00:00:12,180 *rC3 preroll music* 2 00:00:12,180 --> 00:00:17,311 ysf: Hello and welcome to the infrastructure review of the rC3 this 3 00:00:17,311 --> 00:00:22,730 year, 2020. What the hell happened? How could it happen? I'm not alone this year. 4 00:00:22,730 --> 00:00:28,410 With me is lindworm who will help me with the slides and everything else I'm going 5 00:00:28,410 --> 00:00:35,586 to say before. And this is going to be a great fuck up like last year, maybe. We 6 00:00:35,586 --> 00:00:40,683 have more teams, more people, more streams, more of everything. And the first 7 00:00:40,683 --> 00:00:44,934 team and lindworm who I'm going to introduce is the SHOC. Are you there with me? 8 00:00:44,934 --> 00:00:52,140 Lindworm: Oh, yeah, so I got to go to the SHOC. Yeah, it's kind of a stress this 9 00:00:52,140 --> 00:01:00,101 year. We only had about 18 heralds for the main talks rC1 and rC2. And we have 10 00:01:00,101 --> 00:01:05,089 introduced about 51 talks with that. Everybody from this home setup, which was 11 00:01:05,089 --> 00:01:10,020 a very, very hard struggle. So we all had a metric ton of adrenaline and excitement 12 00:01:10,020 --> 00:01:15,750 without… within us. So here you can see what you have seen, how a herald looks 13 00:01:15,750 --> 00:01:25,210 from the front. And so it does look in the background. Oof. That was hard, really 14 00:01:25,210 --> 00:01:32,240 hard for us. So you see all our different set ups here, do we have? And we are very, 15 00:01:32,240 --> 00:01:38,240 very pleased to also have set up a completely new operation center: the 16 00:01:38,240 --> 00:01:46,280 Herald News Show, which I really, really like you to review on YouTube. This was 17 00:01:46,280 --> 00:01:53,909 such a struggle. And we have about, oh, wait a second, so as we said, we're a 18 00:01:53,909 --> 00:01:58,950 little bit unprepared here, I need to have my notes up. There were 20 members that 19 00:01:58,950 --> 00:02:05,690 formed a new team on the first day. They made 23 shows, 10 hours of video 20 00:02:05,690 --> 00:02:13,029 recording, 20 times the pizza man rung at the door. And 23 mate bottles had been 21 00:02:13,029 --> 00:02:19,019 drunk during the preps because all of those people needed to be online the 22 00:02:19,019 --> 00:02:25,090 complete time. So I really applaud to them. That was really awesome, what they 23 00:02:25,090 --> 00:02:30,489 brought over the team and what they brought over the stream. And this is an 24 00:02:30,489 --> 00:02:39,069 awesome team I hope we see more of. ysf, would you take it over? *ysf is muted* 25 00:02:39,069 --> 00:02:46,319 Oh, no. My, my bad. So is the heaven ready? We need to go to the heaven and 26 00:02:46,319 --> 00:02:51,239 would have an infrastructure review of the heaven. 27 00:02:51,239 --> 00:03:29,109 raziel: OK. Du hörst mich noch? Ja, hallo? Ich bin der raziel aus dem Heaven und ehm… 28 00:03:29,109 --> 00:03:38,809 Yeah, heaven is ready, so welcome, everybody. I'm raziel from heaven, and I 29 00:03:38,809 --> 00:03:48,109 will present you the infrastructure review from the heaven team. We had some angel 30 00:03:48,109 --> 00:03:55,620 statistics scrapped out a few hours ago. And on this year, we have not so much 31 00:03:55,620 --> 00:04:04,809 angels like last year, because we had a remote event, but we had a total of 1487 32 00:04:04,809 --> 00:04:17,410 total angels from which 710 arrived and even more of 300 angels that at least 33 00:04:17,410 --> 00:04:28,120 still did one shift. And in total the recorded work done to that point was 34 00:04:28,120 --> 00:04:41,479 roughly 17 and 75 weeks of done working hours, and for the rC3 world we also 35 00:04:41,479 --> 00:04:51,210 prepared a few goodies so people could come visit us. And so we provided them a 36 00:04:51,210 --> 00:05:01,439 few badges there. And every angel that, for example, found our extinguished… 37 00:05:01,439 --> 00:05:09,040 expired extinguisher and also extinguished fire in heaven. The first batch was 38 00:05:09,040 --> 00:05:21,610 achieved from 232 of our angels and even less. But still a good number of 125 39 00:05:21,610 --> 00:05:27,879 angels accomplished to help us and extinguish the fire that broke out during 40 00:05:27,879 --> 00:05:38,090 an event. And with that numbers checked, we also will jump into our heaven. So I 41 00:05:38,090 --> 00:05:45,870 would like to show you some expressions and impressions from it. We had quite the 42 00:05:45,870 --> 00:05:52,830 team working to do exactly what the heaven could do: manage its people so we needed 43 00:05:52,830 --> 00:06:01,320 our heaven office. And we also did this with respect to your privacy, so. We 44 00:06:01,320 --> 00:06:07,259 painted our color… our clouds white as ever, so we cannot see your nicknames, and 45 00:06:07,259 --> 00:06:12,979 you could do your angel work but not be bothered with us asking for your names. 46 00:06:12,979 --> 00:06:22,539 And also, we had prepared some secret passage to our back office. And every time 47 00:06:22,539 --> 00:06:30,330 on the real event, it would happen that some adventurers would find their way into 48 00:06:30,330 --> 00:06:35,050 our back office. And so we needed to provide that opportunity as well, as you 49 00:06:35,050 --> 00:06:42,699 can see here. And let me say that some adventurers tried to find the way in our 50 00:06:42,699 --> 00:06:49,600 sacred digital back office, but only a few were successful. So we hope everyone found 51 00:06:49,600 --> 00:06:58,169 its way back into the real world from our labyrinth. And we also did not spare any 52 00:06:58,169 --> 00:07:07,760 expenses to do some additional update for our angels as well. As you can see, we 53 00:07:07,760 --> 00:07:13,349 tried to do some multi-instance support. So some of our angels also accomplished to 54 00:07:13,349 --> 00:07:21,409 split up and serve more than one angel at a time. And that was quite awesome. And so 55 00:07:21,409 --> 00:07:29,030 we tried to provide the same things we would do on Congress, but now from our 56 00:07:29,030 --> 00:07:39,409 remote offices. And one last thing that doesn't… normally doesn't need to be said. 57 00:07:39,409 --> 00:07:48,060 But I think in this year and with this different kind of event, I think it's 58 00:07:48,060 --> 00:07:55,099 necessary that the heaven as a representative, mostly for people trying 59 00:07:55,099 --> 00:08:05,029 to help make this event awesome. And I think it's time to say the things we do 60 00:08:05,029 --> 00:08:11,610 take for granted. And that is thank you for all your help. Thank you for all the 61 00:08:11,610 --> 00:08:20,259 entities, all the teams, all the participants that achieved the goal to 62 00:08:20,259 --> 00:08:27,289 bring our real Congress that many, many entities missed this year into a new 63 00:08:27,289 --> 00:08:33,570 stage. We tried that online. It had its ups and downs. But I still think it was an 64 00:08:33,570 --> 00:08:40,360 awesome adventure for everyone. And from the Heaven team I can only say thank you 65 00:08:40,360 --> 00:08:47,870 and I hope to see you all again in the future on a real event. Bye! And have a 66 00:08:47,870 --> 00:09:06,519 nice New Year. lindworm: Hello, hello, back again. So we 67 00:09:06,519 --> 00:09:17,500 now are switching over to the Signal Angels. Are the signal angels ready? 68 00:09:17,500 --> 00:09:24,260 Hello! trilader: Yeah, hello, uhm, welcome to the 69 00:09:24,260 --> 00:09:30,350 infrastructure review for the Signal Angels, I have prepared some stuff for 70 00:09:30,350 --> 00:09:36,180 you. This was for us… slides, please? This was for us the first time running a fully 71 00:09:36,180 --> 00:09:49,579 remote Q&A session set, I guess? We had some experience with DiVOC and had gotten 72 00:09:49,579 --> 00:09:54,160 some help from there on how to do this, but just to compare, our usual procedure 73 00:09:54,160 --> 00:09:59,260 is to have a signal angel in the room. They collect the question on their laptop 74 00:09:59,260 --> 00:10:06,000 there and they communicate with the Herald on stage and they have a microphone like 75 00:10:06,000 --> 00:10:09,509 I'm wearing a headset. But in there we have a studio microphone and we speak 76 00:10:09,509 --> 00:10:17,500 questions into it. Yeah, but remotely we really can't do that. Next slide. Because, 77 00:10:17,500 --> 00:10:22,730 well, it would be quite a lot of hassle for everyone to set up good audio setups. 78 00:10:22,730 --> 00:10:29,550 So we needed a new remote procedure. So we figured out that with a signal Angel and 79 00:10:29,550 --> 00:10:34,230 the Herald could communicate via a pad and we could also collect the 80 00:10:34,230 --> 00:10:39,060 question in there. And the Herald will read the question to the speaker and 81 00:10:39,060 --> 00:10:52,779 collect feedback and stuff. So we had 175. No, 157 shifts, and sadly we couldn't fill 82 00:10:52,779 --> 00:11:02,980 five of them in the beginning because there was not enough people already there. 83 00:11:02,980 --> 00:11:07,930 And also technically it was more than five unfilled shifts because for some reasons 84 00:11:07,930 --> 00:11:16,800 there were DJ sets and other things that aren't talks and also don't have Q&A. We 85 00:11:16,800 --> 00:11:21,589 had 61 angels coordinated by four supporters, so me and three other people, 86 00:11:21,589 --> 00:11:26,110 and we had a 60 additional angels that in theory wanted to do signal angel work 87 00:11:26,110 --> 00:11:34,660 but didn't show up to the introduction meeting. Next! For, as I've said for each 88 00:11:34,660 --> 00:11:40,290 session, each talk, we created a pad where we put in the questions from IRC, 89 00:11:40,290 --> 00:11:47,240 Mastodon, and Twitter and. Well, we have a bit more pads than talks we actually 90 00:11:47,240 --> 00:11:53,970 handled, and I have some statistics about an estimated number of questions per talk. 91 00:11:53,970 --> 00:11:59,310 What we usually assume is that there's a question per line, but some questions are 92 00:11:59,310 --> 00:12:03,180 really long and have to split over multiple lines. There are some structured 93 00:12:03,180 --> 00:12:08,089 questions with headings and paragraphs some heralds or signal angels removed 94 00:12:08,089 --> 00:12:12,730 questions after they were done. And also there were some chat and other 95 00:12:12,730 --> 00:12:19,029 communication in there. So next slide, we took a Python script, download all the pad 96 00:12:19,029 --> 00:12:23,220 contents, read them, counted the number of lines, remove the size of the static 97 00:12:23,220 --> 00:12:36,699 header. And in the end we had 179 pads and 1,627 lines if we discount the static 98 00:12:36,699 --> 00:12:42,870 header of nine lines per pad. So that in theory leads to about nine questions in 99 00:12:42,870 --> 00:12:47,610 quotation marks because it's not really questions but lines. But it's an estimate, 100 00:12:47,610 --> 00:12:55,620 per talk. Thank you. ysf: ... talk and what I've learned is 101 00:12:55,620 --> 00:13:03,470 never miss the introduction. So the next in line are the line producers ha ha ha ha 102 00:13:03,470 --> 00:13:32,550 stb are you there? stb: I am here, in fact, so *singing*. So 103 00:13:32,550 --> 00:13:39,170 the people a bit older might recognize this melody badly sung by yours truly and 104 00:13:39,170 --> 00:13:46,029 other members of the line producers team, and I'll get to why that is relevant to 105 00:13:46,029 --> 00:13:53,040 what we've been doing at this particular event. So what does, what do line 106 00:13:53,040 --> 00:13:57,709 producers do? What does an, Aufnahmeleitung actually perform? It's 107 00:13:57,709 --> 00:14:01,339 basically communication between everybody who's involved in the production, the 108 00:14:01,339 --> 00:14:06,529 people behind the camera and also in front of the camera. And so our work started 109 00:14:06,529 --> 00:14:14,009 really early, basically at the beginning of November, taking on like prepping 110 00:14:14,009 --> 00:14:18,449 speakers in a technical setup and rehearsing with them a little bit and then 111 00:14:18,449 --> 00:14:25,089 enabling the studios to allow them to actually do the production coordination on 112 00:14:25,089 --> 00:14:29,490 an organizational side. The technical side was handled by the VOC, and we'll get to 113 00:14:29,490 --> 00:14:36,910 hear about that in a minute. But getting all these people synced up and working 114 00:14:36,910 --> 00:14:42,730 together well, that was quite a challenge. And that took a lot of Mumbles with a lot 115 00:14:42,730 --> 00:14:50,820 of people in them. We only worked on the two main channels. There's quite a few 116 00:14:50,820 --> 00:14:57,690 more channels that are run independently of kind of the central organization. And 117 00:14:57,690 --> 00:15:02,410 again, we'll get to hear about the details of that in a minute. And so we provided 118 00:15:02,410 --> 00:15:06,860 information. We tried to fill wiki pages with relevant information for everybody 119 00:15:06,860 --> 00:15:15,440 involved. So that was our main task. So what does that mean specifically, the 120 00:15:15,440 --> 00:15:24,649 production set up? We had 25 studios, mainly in Germany, also one in 121 00:15:24,649 --> 00:15:32,170 Switzerland. These did produce recordings ahead of time for some speakers, and many 122 00:15:32,170 --> 00:15:39,370 did live set ups for their own channels and also for the two main channels. And 123 00:15:39,370 --> 00:15:43,850 I've listed everybody involved in the live production here. And there were 19 124 00:15:43,850 --> 00:15:49,959 channels in total. So a lot of stuff happening. 25 studios, 19 channels that 125 00:15:49,959 --> 00:15:54,620 broadcast content produced by these studios. So that's kind of the Eurovision 126 00:15:54,620 --> 00:15:59,529 kind of thing, where you have different studios producing content and trying to 127 00:15:59,529 --> 00:16:05,610 mix it all together. Again, the VOC took care of the technical side of things very 128 00:16:05,610 --> 00:16:11,040 admirably, but getting everybody on the same page to actually do this was not 129 00:16:11,040 --> 00:16:18,709 easy. For the talk program, we had over 350 talks in total, 53 in the main channels 130 00:16:18,709 --> 00:16:24,680 And so handling all that, making sure everybody has the speaker information 131 00:16:24,680 --> 00:16:33,149 they need and all these organizational stuff, that was a lot of work. So we 132 00:16:33,149 --> 00:16:38,300 didn't have a studio for the main channels, the 25 studios or the nine, the 133 00:16:38,300 --> 00:16:43,709 live channels, the 12, they actually did provide the production facilities for the 134 00:16:43,709 --> 00:16:49,259 speakers so we can look at the next slide. There's a couple more numbers and of 135 00:16:49,259 --> 00:16:56,029 course, a couple pictures from us working basically from today. We had 53 channel... 136 00:16:56,029 --> 00:17:05,010 53 talks in the main channel. 18 of them were prerecorded and played out. We had 3 137 00:17:05,010 --> 00:17:10,020 where people were actually on location in a studio and gave their talk from there. 138 00:17:10,020 --> 00:17:16,310 And we had 32 that were streamed live like I am speaking to you now with various 139 00:17:16,310 --> 00:17:21,700 technical bits that again the VOC will go into in a minute. And we did a lot of 140 00:17:21,700 --> 00:17:26,070 Q&As, I don't have the numbers how many talks actually had Q&As, but most of them 141 00:17:26,070 --> 00:17:33,261 did, and those were always like. We had a total of 63 speakers we did prepare, at 142 00:17:33,261 --> 00:17:38,750 least the live Q&A session for and helped them set up, we helped them record their 143 00:17:38,750 --> 00:17:44,670 talks if they wanted to prerecord them. So we spent anywhere between one and two 144 00:17:44,670 --> 00:17:49,550 hours with every speaker to make sure they would appear correctly and in good quality 145 00:17:49,550 --> 00:17:55,900 on the screen. And then during the four days, we, of course, helped coordinate 146 00:17:55,900 --> 00:18:00,350 between the master control room and the twelve live studios to make sure that the 147 00:18:00,350 --> 00:18:03,940 speakers were where they were supposed to be and any technical glitches could be 148 00:18:03,940 --> 00:18:09,390 worked out and decide on the spot. If, for example, the line producers made a mistake 149 00:18:09,390 --> 00:18:14,718 and a talk couldn't happen as we had planned because we forgot something. So we 150 00:18:14,718 --> 00:18:19,697 rescheduled and found a new spot for the speakers. So apologies again for that. And 151 00:18:19,697 --> 00:18:24,560 thank you for your understanding and helping us bring you on screen on day two 152 00:18:24,560 --> 00:18:31,473 and not day one. But I'm very glad that that we could work that out. And that's 153 00:18:31,473 --> 00:18:40,130 pretty much it from the line producers, I think. Next up is the VOC. 154 00:18:40,130 --> 00:18:44,923 ysf: Thank you stb. Yes, you're right, the next are the VOC and kunsi and 155 00:18:44,923 --> 00:18:53,870 JW2CAlex are waiting for us. Franzi: ... is Franzi from the VOC. 2020 156 00:18:53,870 --> 00:19:05,150 was the year... Hm? Hi, this is Franzi from the... from VOC. 2020 was the year of 157 00:19:05,150 --> 00:19:12,420 distributed conferences. We had 2 DiVOCs and the FrOSCon to learn how we are going 158 00:19:12,420 --> 00:19:17,380 to produce remote talks. We learned a lot of stuff on organization, Big Blue Button 159 00:19:17,380 --> 00:19:23,760 and Jitsi recording. We had a lot of other events which was just streaming like 160 00:19:23,760 --> 00:19:33,020 business as usual. So for rC3, we extended the streaming CDN with two new locations, 161 00:19:33,020 --> 00:19:41,740 now 7 in total, with a total bandwidth of about 80 gigabits per second. We have two 162 00:19:41,740 --> 00:19:51,380 new mirrors for media.ccc.de and are now also distributing the front end. We got 163 00:19:51,380 --> 00:19:57,860 two new transcoder machines, Erfas and Enhanced cir setup we now have 10 Erfas 164 00:19:57,860 --> 00:20:07,152 with own productions on media.ccc.de. So the question is, will it scale? On the 165 00:20:07,152 --> 00:20:10,070 next slide... Alex: Yeah, next slide. 166 00:20:10,070 --> 00:20:21,435 Franzi: ... we will see that it did scale. We did produce content for 25 167 00:20:21,435 --> 00:20:28,560 studios and 19 channels, so we got lots of lots of recordings which will be published 168 00:20:28,560 --> 00:20:36,190 on media.ccc.de in the next days and weeks. Some have already been published, 169 00:20:36,190 --> 00:20:43,180 so there's a lot of content for you to watch. And now Alex will tell us something 170 00:20:43,180 --> 00:20:47,920 about the technical part. Alex: My name is Alex, Pronouns it/its. I 171 00:20:47,920 --> 00:20:52,300 will now tell you the technical part first, but more of the organization. I was 172 00:20:52,300 --> 00:20:56,907 between the VOC and the line producing team. And now a bit how it worked. So we 173 00:20:56,907 --> 00:21:02,290 had those two main channels, rc-one and rc-two. Those channels have been produced 174 00:21:02,290 --> 00:21:07,740 by the various studios distributed around the whole country. And those streams, 175 00:21:07,740 --> 00:21:12,210 this is now the upper path in the picture, went to our ingest relay, to the FEM, to 176 00:21:12,210 --> 00:21:15,990 the master control room. In Ilmenau there were a team of people adding the 177 00:21:15,990 --> 00:21:20,830 translations, making the mix, making the mixdown, making records and then 178 00:21:20,830 --> 00:21:25,977 publishing it back to the streaming relays. All the other studios produced to 179 00:21:25,977 --> 00:21:30,532 channels. Those channels took the also the signals from different studios, make a 180 00:21:30,532 --> 00:21:36,381 mixdown, etc. publish to our CDN and relays and we publish to the studio 181 00:21:36,381 --> 00:21:40,901 channels. As you can see, this is not the typical setup we had in the last year in 182 00:21:40,901 --> 00:21:47,148 the presence. So, our next slide, we can see where this leads: Lots of 183 00:21:47,148 --> 00:21:53,080 communication. We had the line producing team, we had some production in Ilmenau 184 00:21:53,080 --> 00:21:57,060 that has to be coordinated. We have the studios, we have the local studio helping 185 00:21:57,060 --> 00:22:02,120 Angels. We have some Mumbles there, some RocketChat here, some CDN people some web 186 00:22:02,120 --> 00:22:07,032 where something happens. We have some documentation that should be. And then we 187 00:22:07,032 --> 00:22:12,640 started to plot down the communication paths. Next slide, please. If you plotted 188 00:22:12,640 --> 00:22:16,510 all of them, it really looks like the world, but this is actually the world, but 189 00:22:16,510 --> 00:22:20,490 sometimes it feels like they're just getting lost in different paths. Who you 190 00:22:20,490 --> 00:22:25,107 have to ask, who do you have to call? Where are you? What's the shortest path to 191 00:22:25,107 --> 00:22:33,120 communicate? But let's have a look at the studios. First going to ChaosWest. 192 00:22:33,120 --> 00:22:39,610 Franzi: Yes, on the next slide, you will see the studio set up at ChaosWest TV. So 193 00:22:39,610 --> 00:22:46,782 thank you, ChaosWest for producing your channel. 194 00:22:46,782 --> 00:22:51,130 Alex: At the next slide, you see the Wikipaka television and fernseh-streamen 195 00:22:51,130 --> 00:22:54,922 (WTF) who have the internal motto: "Absolut nicht sendefähig - chaos of 196 00:22:54,922 --> 00:23:00,531 recording". But even then, at some studios, you look more like studios, so 197 00:23:00,531 --> 00:23:08,330 this time at the next slide at the hacc. Franzi: Yeah, at hacc, you will also see 198 00:23:08,330 --> 00:23:14,920 some of the bloopers we had to deal with. So, for example, here you can see there 199 00:23:14,920 --> 00:23:25,270 was a cat in the camera view, so, yeah. And Alex, tell us about the open 200 00:23:25,270 --> 00:23:28,390 infrastructure orbit. Alex: The open infrastructure orbit 201 00:23:28,390 --> 00:23:31,720 showed. In this picture, you can see it's really hard to see how you can make a 202 00:23:31,720 --> 00:23:34,930 studio look really nice, even if you're alone, feeling a bit comfier, more 203 00:23:34,930 --> 00:23:39,988 hackish. But you have also those normal productions as in the next slide. The 204 00:23:39,988 --> 00:23:45,679 Chaosstudio Hamburg Franzi: Yeah, at Chaosstudio Hamburg, we 205 00:23:45,679 --> 00:23:52,555 had two regular work cases like, you know, from all the other conferences, and they 206 00:23:52,555 --> 00:24:02,400 were producing, onsite in a regular studio set up. And last but not least, we got 207 00:24:02,400 --> 00:24:08,220 some impressions from ChaosZone TV. Alex: As you can see here, also quite 208 00:24:08,220 --> 00:24:12,680 regular studio setup, quite regular. No. There was some Corona virus ongoing, and 209 00:24:12,680 --> 00:24:16,530 this is we had a lot of distancing, wearing mask and all the stuff that 210 00:24:16,530 --> 00:24:22,580 everyone is safe but c3yellow (c3gelb) will tell you some facts about it. But 211 00:24:22,580 --> 00:24:28,670 let's look at the nice things. For example, the minor issue: On the second 212 00:24:28,670 --> 00:24:33,864 day, we were sitting there looking at our nice Grafana. Oh, we got a lot of more 213 00:24:33,864 --> 00:24:38,580 connections. The server load's increasing. The first question was: Have we enabled 214 00:24:38,580 --> 00:24:44,570 our cache?". We don't know. But the number of connections is growing that people are 215 00:24:44,570 --> 00:24:50,320 watching our streams, the interest goes up. And we were, well, at least the people 216 00:24:50,320 --> 00:24:56,630 are watching the streams. If there is a website, who cares, the interest works. 217 00:24:56,630 --> 00:25:01,840 But then we suddenly get the relations. Well, something did not really scale that 218 00:25:01,840 --> 00:25:09,820 good. And then using the next slide, this view. This switched pretty fast from after 219 00:25:09,820 --> 00:25:14,595 looking at this traffic graph. "Well, that's interesting" into "Well, we should 220 00:25:14,595 --> 00:25:18,951 investigate". We get thousands of messages on Twitter DMs. We got thousands of 221 00:25:18,951 --> 00:25:23,470 messages in RocketChat, IRC, and suddenly we had a lot of connections to handle; a 222 00:25:23,470 --> 00:25:27,716 lot of inquiries to handle, a lot of phone calls, etc. to handle. And we have to 223 00:25:27,716 --> 00:25:31,110 prioritize for us the hardware then the communication, because otherwise the 224 00:25:31,110 --> 00:25:39,280 information won't stop. On the next slide you can see what our minor issue was. So 225 00:25:39,280 --> 00:25:43,150 at first, we get a lot of connections to our streaming web pages, then to load 226 00:25:43,150 --> 00:25:48,755 balancers, and finally to our DNS servers. A lot of them were quite malformed. It 227 00:25:48,755 --> 00:25:53,906 looked like a storm. But the more important thing we had to deal was all 228 00:25:53,906 --> 00:25:59,694 those passive aggressive messages from, from different persons who said: "Well, 229 00:25:59,694 --> 00:26:04,050 you can't even handle streaming. What are you doing here?" And we worked together 230 00:26:04,050 --> 00:26:08,280 with the c3infra team, thanks for that, how to scale and decentralize a bit more just to 231 00:26:08,280 --> 00:26:13,920 provide the people the connection power they need. So I think in the last years, 232 00:26:13,920 --> 00:26:18,870 we don't need to use more bandwith. We showed we can provide even more bandwith 233 00:26:18,870 --> 00:26:27,170 if we need it. And then, noting everything down… 234 00:26:27,170 --> 00:26:35,990 Franzi: So is it time to shut everything down? No, we won't shut everything down. 235 00:26:35,990 --> 00:26:42,910 The studios can keep their endpoints, can continue to stream on their endpoints as 236 00:26:42,910 --> 00:26:48,030 they wish. We want to keep in touch with you and the studios, produce content with 237 00:26:48,030 --> 00:26:56,520 you, improve our software stack, improve other things like the ISDN, the Internet 238 00:26:56,520 --> 00:27:05,540 Streaming Digital Node, the project for small camera recording setups for sending 239 00:27:05,540 --> 00:27:12,872 to speakers needs developers for the software. Also, KEVIN needs developers and 240 00:27:12,872 --> 00:27:20,530 testers. What's KEVIN? Oh, we have prepared another slide or the next slide. 241 00:27:20,530 --> 00:27:28,450 KEVIN is short for Killer Experimental Video Internet Noise, because we initially 242 00:27:28,450 --> 00:27:36,260 wanted to use OBS.Ninja, but there are a couple of licensing issues. There is not 243 00:27:36,260 --> 00:27:45,190 everything in OBS.Ninja is open source like we wanted, so we decided to code our 244 00:27:45,190 --> 00:27:52,910 own OBS.Ninja-style software. So if you are interested in doing so, please get 245 00:27:52,910 --> 00:28:01,710 into contact with us or visit the wiki. So that's all from the VOC. And we are now 246 00:28:01,710 --> 00:28:10,960 heading over to c3lingo. ysf: Exactly. c3lingo oskar should be 247 00:28:10,960 --> 00:28:23,130 waiting Studio 2, aren't you? 248 00:28:23,130 --> 00:28:28,190 oskar: Yeah, hallo. Hi, yeah, I'm oskar 249 00:28:28,190 --> 00:28:41,210 from c3lingo. We will jump straight into the stats on our slides. As you can see 250 00:28:41,210 --> 00:28:47,920 here, we translated 138 talks this time, as you can see, it's also way less 251 00:28:47,920 --> 00:28:54,430 languages than in the other chaos events that we had since our second languages 252 00:28:54,430 --> 00:28:57,950 team that does everything that is not English and German was only five people 253 00:28:57,950 --> 00:29:02,880 strong this time. So we only managed to do five talks into French and three talks 254 00:29:02,880 --> 00:29:12,730 into Brazilian Portuguese. And then on the next slide… We are looking at our coverage 255 00:29:12,730 --> 00:29:17,320 for the talks and we can see that on the main talks we managed to cover all talks 256 00:29:17,320 --> 00:29:22,480 that were happening from English to German and German to English, depending on what 257 00:29:22,480 --> 00:29:30,350 the source language was. And then, on the other languages track, we only managed to 258 00:29:30,350 --> 00:29:35,151 do 15 percent of the talks from the main channels. And then on the further 259 00:29:35,151 --> 00:29:39,240 channels, which is a couple of others that also were provided to us in the 260 00:29:39,240 --> 00:29:46,410 translation team, we managed to do 68% of the talks, but none of them were 261 00:29:46,410 --> 00:29:52,530 translated into other languages than English and German. On the next slide, 262 00:29:52,530 --> 00:29:58,710 some global stats. We have 36 interpreters, which in total managed to 263 00:29:58,710 --> 00:30:06,454 translate 106 hours and 7 minutes of talks into another language simultaneously. And 264 00:30:06,454 --> 00:30:11,480 the maximum number of hours one person did was 16 hours and the minimum number of 265 00:30:11,480 --> 00:30:16,770 hours, the average number of hours people did was around 3 hours of translation 266 00:30:16,770 --> 00:30:26,970 across the entire event. All right. Then I also have some anecdotes to tell and some 267 00:30:26,970 --> 00:30:31,190 some mentions I want to do. We had two new interpreters that we want to say "hi" to, 268 00:30:31,190 --> 00:30:35,980 and we had a couple of issues with the digital thing that didn't have before with 269 00:30:35,980 --> 00:30:41,100 regular events where people were present. For example, the issue of sometimes when 270 00:30:41,100 --> 00:30:46,160 two people are translating one person's starts interpreting something on wrong 271 00:30:46,160 --> 00:30:50,230 stream. Maybe they were watching the wrong one. And then the partner just thinks they 272 00:30:50,230 --> 00:30:54,230 have more delay or something. Or, for example, a partner having a smaller delay 273 00:30:54,230 --> 00:30:58,530 and then thinking the partner can suddenly read minds because they can translate 274 00:30:58,530 --> 00:31:02,506 faster than the other person is actually seeing the stream. Those are issues that 275 00:31:02,506 --> 00:31:09,180 we usually didn't have with the regular stream, but only with the regular events, 276 00:31:09,180 --> 00:31:17,620 not with remote events. And yeah, some hurdles to overcome. Another thing was, 277 00:31:17,620 --> 00:31:24,390 for example, when on the r3s stage, the audio cut out sometimes for us and but 278 00:31:24,390 --> 00:31:27,851 because one of our translators had also already translated the talk twice, at 279 00:31:27,851 --> 00:31:33,160 least partially to because and it was already canceled after those, they 280 00:31:33,160 --> 00:31:37,630 basically knew most of the content and could basically do a Powerpoint Karaoke 281 00:31:37,630 --> 00:31:43,360 translation and was able to do most of the talk just from the slides without any 282 00:31:43,360 --> 00:31:54,370 audio. Yeah, and then there also was... The last thing I want to say is actually I 283 00:31:54,370 --> 00:31:58,990 wanted to say, give a big shout out to the two of our team members that weren't able 284 00:31:58,990 --> 00:32:02,380 to interpret with us this time because they put their heart and soul into this 285 00:32:02,380 --> 00:32:06,750 event happening. And that's stb and katti, and that's basically everything from 286 00:32:06,750 --> 00:32:16,492 c3lingo. Thanks. 287 00:32:16,492 --> 00:32:29,200 ysf: *muted* 288 00:32:29,200 --> 00:32:37,275 Hello, c3subtitles is it now. td will show the right text to his slides you already 289 00:32:37,275 --> 00:32:48,353 saw a minute ago. td: OK. OK, hi, so I'm td from the 290 00:32:48,353 --> 00:32:54,590 c3subtitles team. And next slide, please. So just to quickly let you know how we get 291 00:32:54,590 --> 00:32:59,690 from the recorded talks to the released subtitles. Well we take the recording 292 00:32:59,690 --> 00:33:04,756 videos and apply speech recognition software to get a raw transcript. And then 293 00:33:04,756 --> 00:33:07,860 Angels work on that transcript to correct all the mistakes that the speech 294 00:33:07,860 --> 00:33:12,620 recognition software makes. And we again apply some autotiming magic to to get some 295 00:33:12,620 --> 00:33:18,070 raw subtitles. And then again Angels do quality control on these tracks to get 296 00:33:18,070 --> 00:33:24,340 released subtitles. Next slide, please. So as you can see, we have various subtitle 297 00:33:24,340 --> 00:33:30,360 tracks in different stages of completion. And these are seconds of material that we 298 00:33:30,360 --> 00:33:35,101 have can see all the numbers are going up and to the right as they should be. So 299 00:33:35,101 --> 00:33:42,280 next slide, please. In total, we had 68 distinct angels that worked 4 shifts on 300 00:33:42,280 --> 00:33:47,290 average. 83 percent of our angels returned for a second shift. 10 percent of our 301 00:33:47,290 --> 00:33:55,110 angels worked 12 or more shifts. And in sum we had 382 hours of angel work for 47 302 00:33:55,110 --> 00:34:01,267 hours of material. So far we've had two releases for rc3 and hopefully more yet to 303 00:34:01,267 --> 00:34:05,590 come, and 37 releases for all the congresses, mostly on the first few days 304 00:34:05,590 --> 00:34:10,590 where we didn't have many recordings. We have 41 hours still in the transcribing 305 00:34:10,590 --> 00:34:16,530 stage of material, 26 hours of material in the timing stage and 51 hours material in 306 00:34:16,530 --> 00:34:21,509 the quality control stage. So there's still lots of work to be done. Next slide, 307 00:34:21,509 --> 00:34:26,206 please. When you have transcripts, you can do fun stuff with them. For example, you 308 00:34:26,206 --> 00:34:31,679 can see that important to people in this talk are "people". We are working on other 309 00:34:31,679 --> 00:34:37,217 cool features that are yet to come. Stay tuned for that. Next slide, please. So to 310 00:34:37,217 --> 00:34:42,250 keep track of all these tasks, we've been using a state-of-the-art high-performance 311 00:34:42,250 --> 00:34:46,737 lock-free NoSQL columnar data store, a.k.a. a kanban board in the previous 312 00:34:46,737 --> 00:34:50,960 years. And because we don't have any windows in the CCL building anymore, we 313 00:34:50,960 --> 00:34:55,799 had to virtualize that. So we're using kanban software now. At this point, I 314 00:34:55,799 --> 00:34:59,861 would like to thank all our hard-working angels for the work. And next slide 315 00:34:59,861 --> 00:35:04,959 please. If you're feeling bored between congresses then you can work on some 316 00:35:04,959 --> 00:35:08,940 transcripts. Just go to c3subtitles.de. If you're interested in our work, follow us 317 00:35:08,940 --> 00:35:14,972 on Twitter. And there's also a link to the release subtitles here. So that's all. 318 00:35:14,972 --> 00:35:23,379 Thank you. ysf: Thank you, td. And before we go into 319 00:35:23,379 --> 00:35:28,990 the POC, where Drake is waiting, I'm sure everyone is asking why are those guys 320 00:35:28,990 --> 00:35:37,869 saying "next slide"? So wait. In the end, we have the infrastructure review 321 00:35:37,869 --> 00:35:44,319 of the infrastructure review meeting going on. So be patient. Now, Drake, are you 322 00:35:44,319 --> 00:35:56,399 ready in Studio 1? 323 00:35:56,399 --> 00:36:00,935 Drake: OK. Hello, I'm Drake from the Phone Operations Center, and 324 00:36:00,935 --> 00:36:04,480 I like to present to you our numbers and maybe some 325 00:36:04,480 --> 00:36:12,293 anecdotes at the end of our part. So please switch to the next slide. Let's get 326 00:36:12,293 --> 00:36:21,028 into the numbers first. So first off, first off, you registered about 1950 ... 327 00:36:21,028 --> 00:36:29,154 5195 sip extensions, which is about 500 more than you registered on the last 328 00:36:29,154 --> 00:36:38,325 congress. Also, you did about 21 000 calls, a little bit less than on the last 329 00:36:38,325 --> 00:36:44,367 congress. But, yeah, we are still quite proud of what you have used our system 330 00:36:44,367 --> 00:36:50,440 with. And yeah, it ran quite stable. And as you may notice on the bottom, we also 331 00:36:50,440 --> 00:36:56,957 had about 23 DECT antennas at the congress or at this event. So please switch to the 332 00:36:56,957 --> 00:37:06,800 next slide. And this is our new feature, it's called the... next slide ..., it 333 00:37:06,800 --> 00:37:11,750 is called the eventphone decentralized DECT infrastructure, which we especially 334 00:37:11,750 --> 00:37:18,839 prepared for this event, the EPDDI. So we had about 23 RFPs online throughout 335 00:37:18,839 --> 00:37:29,510 Germany with 68 DECT telephones of which is up to it. But it's not only the the 336 00:37:29,510 --> 00:37:36,140 German part that we covered. We actually had one mobile station walking out through 337 00:37:36,140 --> 00:37:41,900 Austria, through Passau, I think. So indeed we had an European Eventphone DECT 338 00:37:41,900 --> 00:37:52,119 decentralized infrastructure. Next slide please. We also have some anecdotes, so 339 00:37:52,119 --> 00:37:57,170 maybe some of you have noticed that we had a public phone, a working public phone in 340 00:37:57,170 --> 00:38:03,759 the RC World where you could call other people on the SIP telephone system and 341 00:38:03,759 --> 00:38:10,200 also other people started to play with our system. And I think about yesterday 342 00:38:10,200 --> 00:38:18,759 someone started to introduce c3fire so you could actually control a flame thrower 343 00:38:18,759 --> 00:38:25,111 through our telephone system. And I like to present here a video. Next slide 344 00:38:25,111 --> 00:38:34,650 please. Maybe you can play it. I have quite a delay in waiting for the video to 345 00:38:34,650 --> 00:38:42,670 play. So what you can see here is the c3fire system actually controlled by a 346 00:38:42,670 --> 00:38:54,090 DECT telephone somewhere in Germany. So next slide please. We also provided you 347 00:38:54,090 --> 00:39:04,220 with SSTV servers via the phone number 229, where you could receive some 348 00:39:04,220 --> 00:39:10,220 pictures from event phone, like a postcard basically. So basically you could call the 349 00:39:10,220 --> 00:39:18,961 number and receive a picture or some other pictures, some more pictures. And next 350 00:39:18,961 --> 00:39:28,325 slide please. Yeah basically, that's all from the Eventphone and with that we say 351 00:39:28,325 --> 00:39:34,420 thank you all for the nice and awesome event and yeah, bye from the first 352 00:39:34,420 --> 00:39:43,769 certified assembly POC. Bye. ysf: Thank you, POC, and hello GSM Lynxes 353 00:39:43,769 --> 00:39:51,480 is waiting for us. 354 00:39:51,480 --> 00:39:55,890 lynxes: Yeah, hallo, I'm lynxes, I'm from 355 00:39:55,890 --> 00:40:02,830 the GSM team. This year was quite different as you can imagine. However, 356 00:40:02,830 --> 00:40:11,180 next slide please. So but we managed to get a small network running and also a 357 00:40:11,180 --> 00:40:19,619 couple of SIM cards registering, so where are we now. So next slide please. As you can 358 00:40:19,619 --> 00:40:24,420 see, we are just there in the red dot. There's not even a single line for our 359 00:40:24,420 --> 00:40:32,007 five extensions but we managed 130 calls over five extensions. And next slide 360 00:40:32,007 --> 00:40:44,289 please. So we got, so we got five extensions registered with four SIM cards 361 00:40:44,289 --> 00:40:51,599 and three locations with mixed technologies also two users so far sadly. 362 00:40:51,599 --> 00:40:57,880 And one network with more or less zero problems. And so let's take a look on the 363 00:40:57,880 --> 00:41:06,230 coverage. So next slide please. So we are quite lucky that we managed to get an 364 00:41:06,230 --> 00:41:14,579 international network running. So we got two base stations in Berlin. One in the 365 00:41:14,579 --> 00:41:19,980 hackerspace AfRA, another one north of Berlin. And yeah one of our members is 366 00:41:19,980 --> 00:41:33,200 currently in Mexico. And he's providing the remote chaos networks there. Yes, so 367 00:41:33,200 --> 00:41:43,359 that's basically our network. So before we going to the next slide, we have what we 368 00:41:43,359 --> 00:41:51,490 have done so far is, we are just two people instead of 10 to 20 and had some 369 00:41:51,490 --> 00:41:59,501 fun with improving our network and preparing for the next congress. And next 370 00:41:59,501 --> 00:42:05,813 slide please. And yeah, now I'm closing with the EDGE computing. We improved our 371 00:42:05,813 --> 00:42:15,390 EDGE capabilities and yeah, I wish you a hopefully better year and yeah maybe see 372 00:42:15,390 --> 00:42:22,040 you next year remote or in person. Have fun. 373 00:42:22,040 --> 00:42:30,793 ysf: Thanks and I give a hand to Iindworm for doing the "slide DJ" all the time, and 374 00:42:30,793 --> 00:42:37,020 he now switch to the Haecksen who are next and they bring an image and melzai is 375 00:42:37,020 --> 00:42:46,850 waiting for us in Studio 3. 376 00:42:46,850 --> 00:42:49,720 melzai: Hello, what's phones without people? 377 00:42:49,720 --> 00:42:53,150 So I'm giving now an introduction over here. How many people we needed to 378 00:42:53,150 --> 00:42:58,630 run the whole Haecksen assembly. We had around 20 organizing haecksen and we had 379 00:42:58,630 --> 00:43:03,660 around 20 speakers in our events. And we had in total around 40 events, but I'm 380 00:43:03,660 --> 00:43:09,545 pretty sure that I even don`t know all of these. As you realize, the world is pretty 381 00:43:09,545 --> 00:43:14,999 large. So we needed around seven million pixels to display the whole Haecksen 382 00:43:14,999 --> 00:43:22,785 world. And that needed around 400 commits at our github corner of the internet. 383 00:43:22,785 --> 00:43:28,680 Around 130 people receive the fireplace badge in our case. And around 100 people 384 00:43:28,680 --> 00:43:35,806 tested our swimming pool and received that badge. So great a year for non ???. Also 385 00:43:35,806 --> 00:43:42,930 around 49 people showed some very deep dedication and checked on all memorials at 386 00:43:42,930 --> 00:43:47,329 our Haecksen assembly. Congratulations for that. There were quite a many of these 387 00:43:47,329 --> 00:43:53,499 ones. Our events are run on our BigBlueButton from the Congress and so we had 388 00:43:53,499 --> 00:44:00,349 starting from day 0 no lags and we're able to host up to 133 people in one session. 389 00:44:00,349 --> 00:44:04,748 And that was quite stable. We also introduced four new members around 13 new 390 00:44:04,748 --> 00:44:10,579 Haecksen joinded just for the Congress. And we increased about to the size of 440 391 00:44:10,579 --> 00:44:16,650 Haecksen overall. Also somewhat, we got new Twitter accounts supporting us, so we have 392 00:44:16,650 --> 00:44:22,411 added over 200 more Twitter accounts. And so, you know, our messages are getting 393 00:44:22,411 --> 00:44:28,202 heard. But besides the ritual, we also did some quite physical things. First of all, 394 00:44:28,202 --> 00:44:32,990 we distributed over 50 physical goodie bags to the people with microcontrollers 395 00:44:32,990 --> 00:44:38,637 and self-sewed masks in it, as you can see on the picture. And also sadly, we shopped 396 00:44:38,637 --> 00:44:44,003 so many rC3 Haecksen-themed trunks that they are now out of stock. But they will 397 00:44:44,003 --> 00:44:53,770 be back in January. Thank you. ysf: No, thank you. And I'm going to send 398 00:44:53,770 --> 00:45:00,200 thanks to the Choaspatinnen… Chaospat*innen… who are waiting in Studio 399 00:45:00,200 --> 00:45:11,010 One. Mike: Hi, all this is Mike from the 400 00:45:11,010 --> 00:45:15,740 Chaospat*innen team. We've been welcoming new attendees and underrepresented 401 00:45:15,740 --> 00:45:20,730 minorities to the chaos community for over eight years. We match up our mentees with 402 00:45:20,730 --> 00:45:25,400 experienced chaos mentors. These mentors help their mentees navigate our world of 403 00:45:25,400 --> 00:45:30,250 chaos events. DiVOC was our first remote event and it was a good proof of concept 404 00:45:30,250 --> 00:45:37,559 for rc3. This year, we had 65 amazing mentees and mentors, two in-world 405 00:45:37,559 --> 00:45:43,249 mentee/mentor matchup sessions, one great assembly event hosted by two of our new 406 00:45:43,249 --> 00:45:49,640 mentees, and a wonderful world map assembly built with more than 1337 407 00:45:49,640 --> 00:45:58,490 kilograms of multicolor pixels. Next slide, please. And here's a small part of 408 00:45:58,490 --> 00:46:03,519 our assembly with our signature propeller hat tables. And thank you to the amazing 409 00:46:03,519 --> 00:46:09,091 Chaospat*innen team: fragilant, jali, azriel and lilafish. And to our great 410 00:46:09,091 --> 00:46:13,730 mentees and mentors. We're looking forward to meeting all of the new mentees at the 411 00:46:13,730 --> 00:46:26,340 next chaos event. 412 00:46:26,340 --> 00:46:33,210 lindworm: Yeah, I think that was my call. 413 00:46:33,210 --> 00:46:49,860 So next up, we'll have the, let me see, the c3adventure! Are you ready? 414 00:46:49,860 --> 00:46:53,569 Roang: Hello, my name is Roang Mewp: and I'm Mewp 415 00:46:53,569 --> 00:46:59,380 Roang: and we will talk about the c3adventure, the 2D world, and what we did 416 00:46:59,380 --> 00:47:11,480 to bring it all online. Next slide please. OK, so when we started out, we looked into 417 00:47:11,480 --> 00:47:20,374 how we could bring a Congress-like adventure to the remote experience. And on 418 00:47:20,374 --> 00:47:29,680 October we started with the development and we had some trouble in that we had 419 00:47:29,680 --> 00:47:35,819 multiple upstream merges that gave us some problems. And also due to just Congress 420 00:47:35,819 --> 00:47:40,769 being Congress, or remote experience being a remote experience, we needed to 421 00:47:40,769 --> 00:47:49,044 introduce features a bit late or add features on the first day. So auth was 422 00:47:49,044 --> 00:47:57,819 merged just 4:40 AM in the first day. And on the second day, we finally fixed the 423 00:47:57,819 --> 00:48:03,630 instance jumps – you know, when you walk from one map to the next – we had some 424 00:48:03,630 --> 00:48:08,279 problems there. But on the second day it all went up. And I hope you have all 425 00:48:08,279 --> 00:48:14,809 enjoyed the badges that have finally been updated and brought into the world today. 426 00:48:14,809 --> 00:48:23,059 What does that all mean? Since we started implementing, there have been 400 git 427 00:48:23,059 --> 00:48:28,782 commits in our repository all-in-all, including the upstream merges. But I think 428 00:48:28,782 --> 00:48:35,920 the more interesting stuff is what has been done since the whole thing went live. 429 00:48:35,920 --> 00:48:42,140 We had 200 additional commits, fixing stuff and making the experience better for 430 00:48:42,140 --> 00:48:52,339 you. Next slide. In order to bring this all online, we not only had to think about 431 00:48:52,339 --> 00:48:57,289 the product itself, not only think about the world itself, but we also had to think 432 00:48:57,289 --> 00:49:03,395 about the deployment. The first commit on the deployer, it's a background service 433 00:49:03,395 --> 00:49:10,459 that brings the experience to you, has been done on 26th of November. We started 434 00:49:10,459 --> 00:49:16,279 the first instance, the first clone of the work adventure through this deployer on 435 00:49:16,279 --> 00:49:23,163 8th of December and a couple of days beforehand, I was getting a bit swamped. I 436 00:49:23,163 --> 00:49:26,863 couldn't do all of the work anymore, because I had to coordinate both of the 437 00:49:26,863 --> 00:49:32,280 projects. And so my colleague took over for me, and helped me out a lot. So I'll 438 00:49:32,280 --> 00:49:38,609 give over to him to explain what he did. Mewp: Yeah. So imagine that on Day -5 I 439 00:49:38,609 --> 00:49:46,581 get a message from a friend that, "Hey, help is needed!" So I say, "OK, let's do 440 00:49:46,581 --> 00:49:55,950 it." And Roang tells me that, "OK, so we can spawn a instance and to scale it 441 00:49:55,950 --> 00:50:03,589 somehow and do that." And I spawned the deployer and my music stops. I streamed 442 00:50:03,589 --> 00:50:08,920 music from the internet, and I wondered why did it stop? And I have noticed that, 443 00:50:08,920 --> 00:50:16,150 oh, there are a lot of logs now. Like, a lot. And I have finally Day -4 noticed 444 00:50:16,150 --> 00:50:26,507 that the deployer was spawning copies of itself each few seconds in the log. So 445 00:50:26,507 --> 00:50:32,762 that was the state back then. Since Day -4 until Day 1, we have basically written the 446 00:50:32,762 --> 00:50:44,793 thing. And that's, well… Day 1 we were ready. Well, almost ready. I mean, we have 447 00:50:44,793 --> 00:50:51,319 like 14 instances deployed. And I forgot to mention that, when we were about to 448 00:50:51,319 --> 00:51:00,450 deploy 200 ones at once, it wouldn't work because all of the things would time out. 449 00:51:00,450 --> 00:51:09,210 So we patched things quickly, and 13 o'clock we had our first deployment. This 450 00:51:09,210 --> 00:51:16,600 worked, and everything was fine, and… wait… Why is everybody on one instance? 451 00:51:16,600 --> 00:51:24,259 So, it turns out that we had a bug, not in the deployer, in the app that would move 452 00:51:24,259 --> 00:51:31,470 you from the lobby to the lobby on a different map. So during the first day, we 453 00:51:31,470 --> 00:51:36,319 have we've had a lot of issues of people not seeing each other because they were on 454 00:51:36,319 --> 00:51:45,329 different instances of the lobby. So we are working hard, and… next slide, please, 455 00:51:45,329 --> 00:51:55,630 so we can see that… we are working hard to reconfigure that to bring you together in 456 00:51:55,630 --> 00:52:01,671 the assembly. I think we have succeeded. You can see the population graph on this 457 00:52:01,671 --> 00:52:09,869 slide. The first day was our almost most popular one. And the next day it would 458 00:52:09,869 --> 00:52:23,600 seem, that's OK, not as popular, but we have hit the peak of 1600 users that day. 459 00:52:23,600 --> 00:52:30,309 What else about this? The most popular instance was lobby, of course. The second 460 00:52:30,309 --> 00:52:37,930 most popular instance was hardware hacking area for a while. Then the third, I think. 461 00:52:37,930 --> 00:52:51,279 Next slide please. We have counted, well, first of all, we've had in total about 205 462 00:52:51,279 --> 00:52:57,159 assemblies. The number was increase day- by-day, because people, through the whole 463 00:52:57,159 --> 00:53:04,880 congress, they were working on their maps. For a while, CERT had over a thousand maps 464 00:53:04,880 --> 00:53:11,328 active in their assembly. Which led to the map server crashing. Some of you might 465 00:53:11,328 --> 00:53:19,072 have noticed that. It stopped working quite a few times during Day 3. And they 466 00:53:19,072 --> 00:53:29,200 have reduced the number of maps to 255. And that was fine. At the end of Day 3, I 467 00:53:29,200 --> 00:53:41,779 have counted for about 628 maps, and this is less than the, if, than was available 468 00:53:41,779 --> 00:53:49,799 in reality, because it was the middle of the night (as always), and it was it 469 00:53:49,799 --> 00:53:56,044 wasn't trivial to count them. But in the maps I have found, we have found over two 470 00:53:56,044 --> 00:54:01,800 million used tiles. So that's something you can really explore. I wish I could 471 00:54:01,800 --> 00:54:12,150 have, but deploying this was also fun. Next slide, please. And what… Yeah? 472 00:54:12,150 --> 00:54:17,610 Roang: Just a quick interject. I really want to thank everyone that has put work 473 00:54:17,610 --> 00:54:23,218 into their maps and made this whole experience work. We, we provided the 474 00:54:23,218 --> 00:54:28,599 infrastructure, but you provided the fun. And so I really want to thank everyone. 475 00:54:28,599 --> 00:54:34,369 Mewp: Yeah, the more things happen on the infrastructure, the more fun we have. We 476 00:54:34,369 --> 00:54:42,819 especially don't like to sleep. So we didn't. I basically exchanged with Roang 477 00:54:42,819 --> 00:54:50,422 the way that I slept five hours and during the night and he slept five hours in the 478 00:54:50,422 --> 00:54:57,441 day. And the rest of the time, we were up. The record, though, is incorrect. Roang is 479 00:54:57,441 --> 00:55:05,350 now 30 hours up straight, because the budgets were too important to bring to you 480 00:55:05,350 --> 00:55:14,400 to go to sleep. The thing you see on this graph is undeployed instances. We were 481 00:55:14,400 --> 00:55:19,522 redeploying things constantly. Usually in the form of redeploying half of the 482 00:55:19,522 --> 00:55:24,200 infrastructure at any given time. The way it was developed, you wouldn't have 483 00:55:24,200 --> 00:55:29,150 noticed that. You wouldn't be kicked off your instances, but for a brief period of 484 00:55:29,150 --> 00:55:40,349 time you wouldn't be able to enter any one. But… Next slide. I have been joking 485 00:55:40,349 --> 00:55:46,214 for a few days at the Congress that they have been implementing a sort of 486 00:55:46,214 --> 00:55:50,323 Kubernetes thing, because it's automatically deploy things, and manage 487 00:55:50,323 --> 00:55:57,160 things, and so on. And I have noticed by Day 3 that I have achieved true 488 00:55:57,160 --> 00:56:04,695 enlightenment and true automation, because we have decided to deploy everything at 489 00:56:04,695 --> 00:56:11,455 once at some point. The reason was that we are being DDOSed, and we had to change 490 00:56:11,455 --> 00:56:21,479 something to mitigate that. And so we did that, and everything was fine. But we 491 00:56:21,479 --> 00:56:27,012 made a typo. We made a typo and the deployment failed. And one the deployment 492 00:56:27,012 --> 00:56:39,070 failed, it deleted all the servers. So, yeah, 405 servers got deleted by what I'm 493 00:56:39,070 --> 00:56:47,990 remembering was a single line. So it was brought out automatically, and that wasn't 494 00:56:47,990 --> 00:56:55,195 a problem. It was all fine, but well, to err is human, to automate mistakes is 495 00:56:55,195 --> 00:57:04,073 devops. Next slide? What's important is that these 405 servers were provided by 496 00:57:04,073 --> 00:57:08,410 Hetzner. We couldn't have done that without their infrastructure, without 497 00:57:08,410 --> 00:57:15,720 their cloud. The reason we got up so quickly after this was that the servers 498 00:57:15,720 --> 00:57:21,479 were deleted, but they could have been reprovisioned almost instantly. So the 499 00:57:21,479 --> 00:57:28,190 whole thing took like 10 minutes to get it back up. And, next slide. That's all. 500 00:57:28,190 --> 00:57:38,686 Thank you all for testing our infrastructure, and see you next year. 501 00:57:38,686 --> 00:57:45,930 ysf: Thank you, c3adventure! So this was clearly the first conference that didn't 502 00:57:45,930 --> 00:57:54,358 clap for falling mate bottles! If that's not the thing, maybe we try next year? The 503 00:57:54,358 --> 00:58:03,369 Lounge. And I know I have to ask for the next slide too. The rc3 Lounge artists. 504 00:58:03,369 --> 00:58:09,239 And I was asked to read every country where someone is in, because everyone had 505 00:58:09,239 --> 00:58:17,491 to make the Lounge what it was: an awesome experience. So there were: Berlin, Mexico City 506 00:58:17,491 --> 00:58:26,269 Honduras, London, Zürich, Stockholm, Amsterdam, Rostock, Glasgow, Leipzig, 507 00:58:26,269 --> 00:58:35,635 Santiago de Chile, Prag, Hamburg, Mallorca, Krakow, Tokyo, Philadelphia. 508 00:58:35,635 --> 00:58:45,170 Frankfurt am Main, Köln, Moscow, Taipei Taiwan, Hannover, Shanghai, Seoul… Seoul, 509 00:58:45,170 --> 00:58:54,885 I think, sorry. Vienna, Hong Kong, Karlsruhe and Guatamala. Thank you guys 510 00:58:54,885 --> 00:59:03,260 for making the Lounge. So the next is the Hub and they should be waiting in 511 00:59:03,260 --> 00:59:32,493 Studio Two. *audible echo* 512 00:59:32,493 --> 00:59:35,190 XXX: …software is based on Django. And 513 00:59:35,190 --> 00:59:41,329 it's intended to be used for the next event. The problem is it was a new 514 00:59:41,329 --> 00:59:52,859 software. We had to do a lot of integrations, yeah, live during Day 0. 515 00:59:52,859 --> 01:00:13,650 Well, OK. No. OK, yeah, hi. I'm presenting the Hub, which is a software we wrote for 516 01:00:13,650 --> 01:00:20,310 this conference. Yeah. It's based on different components, all of them are 517 01:00:20,310 --> 01:00:28,170 based on Django. It's intended to be used on future events as well. Our main problem 518 01:00:28,170 --> 01:00:34,171 was it's a new software. We wrote it and, yeah, a lot of the integrations were only 519 01:00:34,171 --> 01:00:41,790 possible on Day 0 or Day 1. And yeah. So even still today on Day 4, we did a lot of 520 01:00:41,790 --> 01:00:47,220 updates, commits to the repository, and even that numbers on the screens are 521 01:00:47,220 --> 01:00:56,019 already outdated again. But yeah, as you could possibly see, we have a lot of 522 01:00:56,019 --> 01:01:02,346 commits all day, night, or all night long. Only a small digit, 6 AM. I am sorry for 523 01:01:02,346 --> 01:01:10,289 that. Next slide, please. And yeah, because the numbers you're quite busy 524 01:01:10,289 --> 01:01:15,020 using the platform, some of these numbers on the screen are already outdated again. 525 01:01:15,020 --> 01:01:24,935 Out of the 360 assemblies which were registered, only 300 got accepted. Most of 526 01:01:24,935 --> 01:01:32,788 them were, yeah, event or people wanting to do a workshop and trying to register an 527 01:01:32,788 --> 01:01:39,730 assembly. Or, duplicates. So, please organize yourself. Events, currently we have over 528 01:01:39,730 --> 01:01:46,529 940 in the system. You're still clicking events, nice. Thanks for that. The events 529 01:01:46,529 --> 01:01:52,970 are coordinating with the studios, so we are integrating all of the events of all 530 01:01:52,970 --> 01:01:59,156 the studios, and the individual ones, and the self organized sessions. All of them. A new 531 01:01:59,156 --> 01:02:08,180 feature, the badges. Currently you have created 411. And, yeah, from these badges 532 01:02:08,180 --> 01:02:17,830 redeemed, we have 9269 achievements and 19 000 stickers. Documentation, sadly, was 533 01:02:17,830 --> 01:02:26,569 a 404, because yeah. We were really busy doing stuff. Some documentation has 534 01:02:26,569 --> 01:02:33,489 already been written, but yeah. More documentation is, will become available 535 01:02:33,489 --> 01:02:39,859 later. We will open source the whole thing of course, but right now we're still in 536 01:02:39,859 --> 01:02:45,990 production and cleaning up things. And yeah. Finally, for some numbers. Total 537 01:02:45,990 --> 01:02:54,219 requests per second were about 400. In the night, when the world was redeploying, 538 01:02:54,219 --> 01:03:01,359 then we only had about 50 requests per second, but it maxed up to 700 requests 539 01:03:01,359 --> 01:03:08,530 per second. And the authentication for the world, for the 2D adventure, it was about 540 01:03:08,530 --> 01:03:16,640 220 requests per second. More or less stable due to some bugs and due to some 541 01:03:16,640 --> 01:03:23,460 heavy usage. So, yeah, we appreciate that you used the platform, used the new Hub, 542 01:03:23,460 --> 01:03:34,939 and hope to see you on the next event. Thanks. 543 01:03:34,939 --> 01:03:41,744 ysf: Hello Hub. Thank you Hub. And the next is betalars waiting for us. He's from 544 01:03:41,744 --> 01:03:53,380 the c3auti team, and he will tell us what he does and his team did this year. 545 01:03:53,380 --> 01:04:04,119 betalars: Hi, I'm betalars from c3auti, and we've been really busy this year as 546 01:04:04,119 --> 01:04:15,193 you can probably see by the numbers on my next slide. We have 37 confirmed Auti-Angles 547 01:04:15,193 --> 01:04:24,519 and today we surpassed the 200 hours mark. We have 10 Orga Mumbles 548 01:04:24,519 --> 01:04:30,170 leading up to the event and there are almost five million unique pixels in our 549 01:04:30,170 --> 01:04:37,400 repository. I'm pretty convinced we've managed to create the smallest Fairydust 550 01:04:37,400 --> 01:04:45,140 of rC3, provided by an actual space engineer. And the Tree of Solitude is not 551 01:04:45,140 --> 01:04:52,150 the only thing we've managed to create, contribute to this wonderful experience. 552 01:04:52,150 --> 01:05:01,849 On our next slide, you can see that we also contributed six panel sessions for 553 01:05:01,849 --> 01:05:08,260 autistic creatures to discuss their experiences and five Play sessions for 554 01:05:08,260 --> 01:05:18,249 them to socialize. We helped to contribute a talk, a podcast, and an external panel 555 01:05:18,249 --> 01:05:26,099 to the big streams. And on our own panels, we've had up to 80 participants that need 556 01:05:26,099 --> 01:05:32,650 to be split up to five breakout rooms so they could all have a meaningful 557 01:05:32,650 --> 01:05:45,390 discussion. And all their ideas and thoughts were anonymized and stored on more than 1000 558 01:05:45,390 --> 01:05:54,780 lines of markdown documentation that you can find on the Internet. But 1000 lines of 559 01:05:54,780 --> 01:06:00,950 markdown wouldn't be enough for me to express the gratitude I have towards all 560 01:06:00,950 --> 01:06:08,870 the amazing creatures that helped us make this experience happen and for all the 561 01:06:08,870 --> 01:06:17,170 amazing teams that worked with us. I'm so happy to see you again soon, but now I 562 01:06:17,170 --> 01:06:25,639 think I will need some solitude for myself. 563 01:06:25,639 --> 01:06:32,125 ysf: Thank you betalars. So, lindworm, are you ready? The next one is the video, as 564 01:06:32,125 --> 01:06:46,116 far as I know. It's from the C3 Inclusion Operation Center. I don't know the short 565 01:06:46,116 --> 01:06:55,428 name; C3IOC? And it's counting down three two one go. 566 01:06:55,428 --> 01:07:18,921 *video without audio* 567 01:07:18,921 --> 01:07:26,009 So, video is like a very difficult thing to play in those days, because we only used to do 568 01:07:26,009 --> 01:07:33,146 stuff live. Live means a lot of pixels and traffic is done from this here, from this 569 01:07:33,146 --> 01:07:40,047 glass, to all the wires and cables and back to the glass of your screen. And this 570 01:07:40,047 --> 01:07:46,986 is like magic to me, somehow. Although, I. am only. being. a robot. to talk. 571 01:07:46,986 --> 01:07:57,809 synchronistically. with all the.... It's been around enough time, I think, to 572 01:07:57,809 --> 01:08:04,641 switch back to Lindy with the video. I tell you what we are you going to… 573 01:08:04,641 --> 01:08:17,660 *video without audio* 574 01:08:17,660 --> 01:08:23,322 nwng: Hello everyone, I'm nwng from the new C3 Inclusion Operation Center. This 575 01:08:23,322 --> 01:08:27,440 year, we've been working on accessibility guides that help the organizing teams and 576 01:08:27,440 --> 01:08:32,930 assemblies improve the event for everyone, and especially people with disabilities. 577 01:08:32,930 --> 01:08:36,390 We have also worked with other teams individually to figure out what can still 578 01:08:36,390 --> 01:08:40,470 be improved in their specific range of functions - but there are still a lot to 579 01:08:40,470 --> 01:08:45,424 catch up on! Additionally, we have published a completely free and accessible 580 01:08:45,424 --> 01:08:50,800 CSS design template that features dark mode and an accessible font selection. And 581 01:08:50,800 --> 01:08:56,273 it still looks good without Javascript. 100 Internet points for that! For you 582 01:08:56,273 --> 01:08:59,940 visitors, we have been collecting your feedback through mail or twitter – and 583 01:08:59,940 --> 01:09:03,934 won't stop after the Congress! If you stumbled across some barriers, please get 584 01:09:03,934 --> 01:09:11,262 in touch via c3ioc.de or @c3inclusion on twitter to tell us about your findings! 585 01:09:11,262 --> 01:09:19,180 Thanks a lot for having us. ysf: Thank you for the video. Finally, 586 01:09:19,180 --> 01:09:27,670 technical's working! We should… does someone know computers? Maybe? Kritis is 587 01:09:27,670 --> 01:09:33,113 one of them, and he is waiting in Studio One to tell us something about C3 Yellow 588 01:09:33,113 --> 01:09:44,880 or c3gelb wie wir hier sagen. 589 01:09:44,880 --> 01:09:46,540 Kritis: Yeah, welcome. I'm still looking 590 01:09:46,540 --> 01:09:50,470 at this hard drive. Maybe you remember this from the very beginning? It has to be 591 01:09:50,470 --> 01:09:55,750 disinfected really thoroughly, and I guess I can take it out by the end of the event. 592 01:09:55,750 --> 01:10:05,010 And for… the next slide with the words, please. We did found roughly 0777 hands 593 01:10:05,010 --> 01:10:12,770 wash options and 0x3FF waste disposal possibilities. We checked the correct date 594 01:10:12,770 --> 01:10:22,076 on almost all of the 175 disinfectant options you had around here. And because 595 01:10:22,076 --> 01:10:27,390 at a certain point of time, people from CERT were not reachable in the CERT room 596 01:10:27,390 --> 01:10:30,654 because they were running around everywhere else in this great 2D world. We 597 01:10:30,654 --> 01:10:33,996 had the chance to bypass and channel all the information because there were two 598 01:10:33,996 --> 01:10:39,610 digital cats on a digital tree. And so we got the right help to the right option. 599 01:10:39,610 --> 01:10:45,210 Next slide, please. We have a couple of options ongoing. A lot of work had been 600 01:10:45,210 --> 01:10:51,180 done before. We had all the studios with all the corona things going on before, but 601 01:10:51,180 --> 01:10:58,070 now we think we should really watch into an angel disinfectant swimming basin for 602 01:10:58,070 --> 01:11:04,140 the next time, to have there the maximum option of cleanliness. And we will talk 603 01:11:04,140 --> 01:11:10,540 with the BOC. If we can maybe achieve to use this Globuli maxi-cubes for the 604 01:11:10,540 --> 01:11:17,370 Tschunk in the upcoming time. Apart from that, in order to get more Bachblüten and 605 01:11:17,370 --> 01:11:24,330 everything else, we need someone who is able to help us with the Potenzieren 606 01:11:24,330 --> 01:11:31,961 for homoeopathic substances. So if you feel welcome with that, please just drop 607 01:11:31,961 --> 01:11:40,173 us a line to: info@c3gelb.de. Thank you very much and good luck. 608 01:11:40,173 --> 01:11:44,890 ysf: Thank you Kritis. Finally happy to hear your voice. I only know you from 609 01:11:44,890 --> 01:11:50,700 Twitter, where we treat our stuff together, or I yours and you, mine, don't. 610 01:11:50,700 --> 01:11:56,885 Maybe you're going to change it… please? And, talking about messages. Chaos Post 611 01:11:56,885 --> 01:12:06,150 was here too, and trilader, whom we already heard earlier, has more to say. 612 01:12:06,150 --> 01:12:11,350 trilader: OK, welcome. It's me again. I've changed outfits a bit. I'm not here for 613 01:12:11,350 --> 01:12:16,050 the Signal Angels anymore, but for Chaos Post. So, yeah. We had an online office 614 01:12:16,050 --> 01:12:22,540 this year again, as we had with the DiVOCs before. And I've got some mail numbers for 615 01:12:22,566 --> 01:12:28,984 you that should be on the screen right now. If it's not, if it's on the title 616 01:12:28,984 --> 01:12:37,982 page, please switch to the first one where it lists a lot of numbers. We had 576 617 01:12:37,982 --> 01:12:46,370 messages delivered total. This is numbers from around half to six. And 12 of them we 618 01:12:46,370 --> 01:12:51,210 weren't able to deliver because, well, non-existent mailboxes or full mailboxes 619 01:12:51,210 --> 01:12:58,990 mostly. We delivered mail to 43 TLDs, the most going to Germany, to .de domains, 620 01:12:58,990 --> 01:13:06,060 followed by .com, .org, .net, and to Austria with .at; We had a couple of 621 01:13:06,060 --> 01:13:11,810 motifs you could choose from, the most popular one was "Fairydust at Sunset", 95 622 01:13:11,810 --> 01:13:18,050 people selected that. Next slide. About our service quality. We had a minimum 623 01:13:18,050 --> 01:13:25,300 delay from the message coming in, us checking it, and it going out for about a 624 01:13:25,300 --> 01:13:29,643 bit more than four seconds. The maximum delay was about seven hours. That was 625 01:13:29,643 --> 01:13:36,220 overnight, when no agents were ready, or they were all asleep, or having… being 626 01:13:36,220 --> 01:13:40,660 busy with, I don't know, the Lounge or something? And on average a message took 627 01:13:40,660 --> 01:13:47,090 you, took us 33 minutes from you putting it into our mailbox to it getting out. 628 01:13:47,090 --> 01:13:52,620 Some fun facts: We had issues delivering to T-Online at the first two days, but we 629 01:13:52,620 --> 01:13:57,690 managed to get that fixed. A different mail provider refused our mail because it 630 01:13:57,690 --> 01:14:05,170 contained the string c3world, the domain in the mail text. And apparently new 631 01:14:05,170 --> 01:14:08,896 domains are scary, and you can't trust them or something. We had created a ticket 632 01:14:08,896 --> 01:14:14,833 with them, they fixed it, and it was super fast, super nice service. Yeah. Also, some 633 01:14:14,833 --> 01:14:21,090 people tried to sent digital postcards to Mastodon accounts because they looked like 634 01:14:21,090 --> 01:14:26,090 email addresses or something. Another thing that's not on a slide is we had 635 01:14:26,090 --> 01:14:31,870 another new feature this time. That was our named recipients. So you could, for 636 01:14:31,870 --> 01:14:39,850 example, send mail to CERT without knowing their address. And they also have a really 637 01:14:39,850 --> 01:14:44,041 nice postcard wall, where you can see all the postcards you sent them. The link for 638 01:14:44,041 --> 01:14:55,871 that is on our Twitter. Thank you. ysf: Thank you Chaos Post. lindworm, are 639 01:14:55,871 --> 01:14:59,579 you there? lindworm: Ja, ja. Ich bin da, Ich bin da. 640 01:14:59,579 --> 01:15:04,920 Hallo, you're hearing me? ysf: I hear you. 641 01:15:04,920 --> 01:15:12,795 lindworm: So I have to switch some more. It's kind of stressy for me, really. 642 01:15:12,795 --> 01:15:21,370 ysf: You're doing an awesome job. Thank you for doing it. So just out of 643 01:15:21,370 --> 01:15:27,760 curiosity, and did you have a problem accepting any cookies or so? 644 01:15:27,760 --> 01:15:35,500 lindworm: No, not really. ysf: I heard somewhere. That some really 645 01:15:35,500 --> 01:15:39,030 smart people had problems using the site because of cookies. 646 01:15:39,030 --> 01:15:44,800 lindworm: Oh, no, that was not my problem. I only couldn't use the site because of 647 01:15:44,800 --> 01:15:54,190 overcrowding. That was often one of my my little problems. And please, I hope you 648 01:15:54,190 --> 01:15:58,750 don't see what I'm doing right now in the background with starting our pets and so 649 01:15:58,750 --> 01:16:11,820 on. And what I wanted to say to all of you, this was the first Congress where we 650 01:16:11,820 --> 01:16:18,850 have so many women and so many non-cis people running that show and being up 651 01:16:18,850 --> 01:16:25,343 front the camera and making everything up. I would really thank you all. Thank you, 652 01:16:25,343 --> 01:16:30,701 that you made that possible. And thank you that we get more and more diverse, year by 653 01:16:30,701 --> 01:16:38,580 year. ysf: I can only second that. And now we 654 01:16:38,580 --> 01:16:43,100 are switching to C3 Infrastructure. lindworm: Yeah, we need to. 655 01:16:43,100 --> 01:16:50,000 ysf: I'm sure a lot of questions will be answered by them. 656 01:16:50,000 --> 01:16:58,210 lindworm: And I try to make up the slides for that, but I do not find them right 657 01:16:58,210 --> 01:17:02,680 now. patrick: Look mom, I'm on TV. 658 01:17:02,680 --> 01:17:11,044 thies: Yeah. Welcome to the infrastructure review of the Team Infrastructure. I'm not 659 01:17:11,044 --> 01:17:15,960 quite sure if we have the newest revision of the slides, cause my version of the 660 01:17:15,960 --> 01:17:21,890 stream isn't loading right now. So maybe lindworm, is it possible to press 661 01:17:21,890 --> 01:17:30,380 control-R? And you're seeing a burning computer, then we have the actual slides. 662 01:17:30,380 --> 01:17:35,470 Patrick: Let's just Powerpoint Karaoke without the background music. 663 01:17:35,470 --> 01:17:44,000 thies: Yeah, and without the PowerPoint presentation in realtime. Now I'm seeing 664 01:17:44,000 --> 01:17:48,070 me. Let's wait a few seconds until we see a slide. 665 01:17:48,070 --> 01:17:51,800 Patrick: We want to wait the entire stream delay. 666 01:17:51,800 --> 01:18:00,190 thies: It's just about 30 to one minute. Patrick: Well done. 667 01:18:00,190 --> 01:18:09,960 thies: Yeah, I'm thies and I'm waiting. And this is Patrick, and he's waiting too. 668 01:18:09,960 --> 01:18:19,850 Yeah, but that's in the middle of the slides. Can we go… OK. Yeah. I'm now 669 01:18:19,850 --> 01:18:26,799 seeing something in the middle of the slides, but it seems fine. OK, yeah. We 670 01:18:26,799 --> 01:18:36,640 are the team C3 Infra. rC3 Infra. We are creating the infrastructure. Next slide. 671 01:18:36,640 --> 01:18:50,230 We had about nine terabytes of RAM and 1,700 CPU cores. The whole event there's 672 01:18:50,230 --> 01:18:58,023 only one dead SSD that died because everything's broken. We had five dead RAID 673 01:18:58,023 --> 01:19:02,520 controllers, and didn't bother to replace the RAID controllers, just replaced them 674 01:19:02,520 --> 01:19:14,120 with new servers. And 100 percent uptime. Next slide. We looked about 42 hours on 675 01:19:14,120 --> 01:19:22,980 starting screens of enterprise servers. 20 minutes max is what HP delivered. And we 676 01:19:22,980 --> 01:19:32,330 are now certified enterprise observers. We had only 27%-ish of visitors using IPv6. 677 01:19:32,330 --> 01:19:39,540 So that's even less than Google publishes. And even though we had almost full IPv6 678 01:19:39,540 --> 01:19:48,010 coverage – except some really, really shady out-of-band management networks – we're 679 01:19:48,010 --> 01:19:55,370 still not at the IPv6 coverage that we are hoping for. I'm not quite sure if that's 680 01:19:55,370 --> 01:20:05,020 the right slides. But I'm not quite sure where we are in the text. Yeah, Patrick. 681 01:20:05,020 --> 01:20:11,290 Patrick: Yeah, so before the Congress there was one prediction: there's no way 682 01:20:11,290 --> 01:20:17,737 it cannot be not DNS. And while it was DNS at least once, so we checked that box. And 683 01:20:17,737 --> 01:20:27,090 let's go over to the next topic, OS. We provisioned about 300 nodes, and it was an 684 01:20:27,090 --> 01:20:33,181 Ansible-powered madness. So, yeah, there was full disk encryption on all nodes. No 685 01:20:33,181 --> 01:20:37,621 IP logged in the access logs, we took extra care of that. And we configured 686 01:20:37,621 --> 01:20:43,470 minimal logging wherever possible, so the case of some problems we only had WARNINGs 687 01:20:43,470 --> 01:20:50,533 available. And there are no INFO logs, no DEBUG logs; just the minimal logging 688 01:20:50,533 --> 01:20:55,850 configuration. And with some software, we had to pipe logs to /dev/null because the 689 01:20:55,850 --> 01:21:01,080 software just wouldn't stop logging IP's, and we didn't want that. So no personal 690 01:21:01,080 --> 01:21:06,897 data in logs, so no GDPR headache, and your data is safe with us. The Ansible 691 01:21:06,897 --> 01:21:12,030 madness I've talked about was a magical deployment that deep bootstrapped into the 692 01:21:12,030 --> 01:21:18,370 live system and assimilated into the rC3 infrastructure while it's still running. 693 01:21:18,370 --> 01:21:27,430 So if you didn't boot your machine then what? They're just running. When a OS 694 01:21:27,430 --> 01:21:31,529 deployment was broken, it was almost always due to a network or routing. At 695 01:21:31,529 --> 01:21:37,320 least the OS team claims that, and this claim is disputed by the network team of 696 01:21:37,320 --> 01:21:42,723 course. One time, the deployment broke because of a trigger happy infra angel. 697 01:21:42,723 --> 01:21:52,450 But let's not talk about that. Of course, at this point, we want to announce our 698 01:21:52,450 --> 01:21:58,370 great cooperation with our gold sponsor ddos24.net, who provided an excellent 699 01:21:58,370 --> 01:22:05,750 service of handcrafted request to our infrastructure. That was a great demand or 700 01:22:05,750 --> 01:22:14,080 great public demand, with a million requests per second for a while. But even 701 01:22:14,080 --> 01:22:21,540 during the highest or peak demand, we were able to serve most of these services. We 702 01:22:21,540 --> 01:22:27,920 provide the infrastructure to the VOC, and they quickly made use of the provided 703 01:22:27,920 --> 01:22:35,616 infrastructure deployed there. Overall, an amazing time to market. We had six 704 01:22:35,616 --> 01:22:41,254 locations, and those six locations where some wildly different, special snowflakes 705 01:22:41,254 --> 01:22:49,381 overall. So we had Düsseldorf, 816 CPU cores there, two terabytes of RAM, and we 706 01:22:49,381 --> 01:22:55,200 had 10 gigabits per second interconnect. There was also a 1 terabit per second 707 01:22:55,200 --> 01:22:59,670 Infiniband available, but sadly, we couldn't use that. It would have been 708 01:22:59,670 --> 01:23:05,351 nice. The machines that had a weird and ancient IPMI, which made it hard to deploy 709 01:23:05,351 --> 01:23:10,459 there. And the admin on location never deployed bare metal hardware to a 710 01:23:10,459 --> 01:23:15,330 datacenter, so there were also some learning experience there. Fun fact about 711 01:23:15,330 --> 01:23:20,737 Düsseldorf, this was the data center with the maximum heat. One server, seven units, 712 01:23:20,737 --> 01:23:29,770 over 9000 watts of power. 11.6 to be exact. Which is why they had some to take 713 01:23:29,770 --> 01:23:39,882 some creative heat management solutions. Next was Frankfurt, there we had 620 714 01:23:39,882 --> 01:23:47,890 gigabits of total uplink capacity, and we actually only used 22 gigabit during peak 715 01:23:47,890 --> 01:23:54,430 demand. Again, by our premium sponsor: ddos24.net. There was zero network 716 01:23:54,430 --> 01:24:02,690 congestion and 1.5 gigabits per second were IPv6. So there was no real traffic 717 01:24:02,690 --> 01:24:08,980 challenge. For the network engineers of you, it was a full Layer 3 architecture 718 01:24:08,980 --> 01:24:15,960 with MPLS between the WAN routers. And there was a night shift on the 26the and 719 01:24:15,960 --> 01:24:25,400 27th for more servers, because some shipments didn't arrive yet. The fun fact 720 01:24:25,400 --> 01:24:30,346 about this datacenter was the maximum bandwidth. Some servers there had 50 721 01:24:30,346 --> 01:24:36,727 gigabit uplink on the server configured. It was the data center with the maximum 722 01:24:36,727 --> 01:24:41,520 manual intervention. Of course, we had the most infrastructure there and it wasn't 723 01:24:41,520 --> 01:24:47,927 oversubscribed at any point. We had some hardware in Stuttgart, which was basically 724 01:24:47,927 --> 01:24:53,080 the easiest deployment. There were also some night shifts, but the thanks to 725 01:24:53,080 --> 01:24:58,710 neuner and team this was a really easy deployment. It was also the most silent 726 01:24:58,710 --> 01:25:06,590 DC, so no incident from Day -5 until now. So if you're currently watching from 727 01:25:06,590 --> 01:25:13,841 Stuttgart now, you can create some issues because now we said it. Wolfsberg was the 728 01:25:13,841 --> 01:25:18,420 smallest DC. We only had three servers and we managed to kill one hardware RAID 729 01:25:18,420 --> 01:25:25,826 controller, so we only could use two servers there. So, yeah. And then Hamburg 730 01:25:25,826 --> 01:25:30,982 was the data center with the minimum uptime. We never could deploy to this data 731 01:25:30,982 --> 01:25:35,490 center because there was a broken netboot and we couldn't provision anything there. 732 01:25:35,490 --> 01:25:42,210 And of course, the sixth data center was the Hetzler Cloud, where we deployed it on 733 01:25:42,210 --> 01:25:48,220 all locations. Deployment fun facts: we received a covid warning from the data 734 01:25:48,220 --> 01:25:52,790 center. Luckily, it didn't affect us. It was at another location. But thanks for 735 01:25:52,790 --> 01:25:59,710 the heads-up and the warning. The team leader of a sponsor needed to install 736 01:25:59,710 --> 01:26:07,003 Proxmox in a DC with no knowledge, without any clue what they were doing. We 737 01:26:07,003 --> 01:26:11,400 installed Proxmox in the Hamburg DC, and no server actually wanted to talk to us, 738 01:26:11,400 --> 01:26:17,000 so we had to give up on that. And there had to be a lorry relocated before we 739 01:26:17,000 --> 01:26:27,456 could deploy other servers. So that's that was standing in the way there. Now, let's 740 01:26:27,456 --> 01:26:34,877 get to Jitsi. Our peak count were 1,105 users at the same time, on the same 741 01:26:34,877 --> 01:26:41,670 cluster. I don't know if it was at the same time as the peak user count, but the 742 01:26:41,670 --> 01:26:44,500 peak conference count was 204 conferences. 743 01:26:44,500 --> 01:26:49,850 I hope we can still beat that today, but this is data from 744 01:26:49,850 --> 01:26:58,810 yesterday. The peak conference size was 94 participants in a single conference. And 745 01:26:58,810 --> 01:27:06,780 let me give condolences to your computer, because that must have been hard on it. 746 01:27:06,780 --> 01:27:14,171 Our peak outgoing video traffic on the Jitsi video bridges was 1.3 gigabits per 747 01:27:14,171 --> 01:27:23,715 second. And we had about three quarters of the participants were streaming video and 748 01:27:23,715 --> 01:27:31,750 one quarter of them had video disabled. Interesting ratio. Our Jitsi deployment 749 01:27:31,750 --> 01:27:38,140 was completely automated with Ansible, so it was zero to Jitsi in 15 minutes. We 750 01:27:38,140 --> 01:27:43,390 broke up the Jitsi cluster into four shards to have better scalability and 751 01:27:43,390 --> 01:27:48,360 resilience. So if one shard went down, it would only affect part of the conferences 752 01:27:48,360 --> 01:27:53,380 and not all of them. Because there are some infrastructure components that you 753 01:27:53,380 --> 01:28:00,495 can't really scale or cluster, so we went with with the sharding route. Our Jitsi 754 01:28:00,495 --> 01:28:07,640 video bridges were at about 42% peak usage – excluding our smallest video bridge, 755 01:28:07,640 --> 01:28:11,400 which was only eight cores and eight gigabytes, which we added in the beginning 756 01:28:11,400 --> 01:28:17,120 to test some stuff out, and it remained in there. And yes, we overprovisioned a bit. 757 01:28:17,120 --> 01:28:21,780 There will also be a blog post on our Jitsi Meet deployment coming in the 758 01:28:21,780 --> 01:28:31,110 future. And for the next time we, for the upcoming days, we will enable 4K streaming 759 01:28:31,110 --> 01:28:40,740 on there. So why not use that? And we want to say thanks to the FFMEET Projekt, who 760 01:28:40,740 --> 01:28:46,400 contacted us after our initial load test and gave us some tips to handle load 761 01:28:46,400 --> 01:28:58,190 effectively and so on. We also tried making DECT call-out working. Spent 48 762 01:28:58,190 --> 01:29:06,760 hours trying to get it to work, but there were some troubles there. So sadly, no 763 01:29:06,760 --> 01:29:14,610 adding DECT participants to your Jitsi conferences for now. jitsi.rc3.world will 764 01:29:14,610 --> 01:29:22,980 be running over New Year. So you can use that to get together with your friends and 765 01:29:22,980 --> 01:29:27,829 so on over the New Year. Stay separate, don't visit each other please. Don't 766 01:29:27,829 --> 01:29:35,920 contribute to covid-19 spread. You've got the alternative there. Now let's go over 767 01:29:35,920 --> 01:29:40,960 to monitoring. thies. thies: Yeah, thanks. First of all, it's 768 01:29:40,960 --> 01:29:46,660 really funny how you edit this page, but reveal.js doesn't work that way until 769 01:29:46,660 --> 01:29:51,824 lindworm reloads the page, which hopefully doesn't do right now. Everything's fine, 770 01:29:51,824 --> 01:29:58,410 so you can leave it to be. Yeah, monitoring. We had to Prometheus and 771 01:29:58,410 --> 01:30:04,920 Alertmanager set up completely driven out of our solemnly one and only source of 772 01:30:04,920 --> 01:30:14,347 truth: our Netbox. We received about 34 858 critical alerts. It's – looking at 773 01:30:14,347 --> 01:30:21,340 my mobile phone – it's definitely more right now. And about 13,070 warnings. Also 774 01:30:21,340 --> 01:30:30,456 definitely more right now. And we tended about 100 of them. The rest was kind of 775 01:30:30,456 --> 01:30:42,494 useless. Next slide, please. As it's important to have an abuse hotline and an 776 01:30:42,494 --> 01:30:48,133 abuse contact, we received two network abuse messages, both from Hetzner – one of 777 01:30:48,133 --> 01:30:51,950 our providers – letting us know that someone doesn't like our infrastructure as 778 01:30:51,950 --> 01:31:01,730 much as we do. Props to ddos24.net. And we got one call it our abuse hotline, and it 779 01:31:01,730 --> 01:31:09,190 was one person who wanted to buy a ticket from us – Sadly, we were out of tickets. 780 01:31:09,190 --> 01:31:16,020 Next slide, please. Some other stuff. We got a premium Ansible deployment brought 781 01:31:16,020 --> 01:31:26,050 to you by turing-complete YAML. That sounds scary. And we had about 130k DNS updates 782 01:31:26,050 --> 01:31:32,291 thanks to the World team. At this point they're really stressing our DNS API with 783 01:31:32,291 --> 01:31:39,210 the re-deployments. And also our DNS, Prometheus, and Grafana are deployed on 784 01:31:39,210 --> 01:31:48,025 and by NixOS thanks to flüpke and head over to flüpkes interweb thingy. He wrote 785 01:31:48,025 --> 01:31:54,706 some blog posts about how to deploy stuff with his NixOS. And the next slide, 786 01:31:54,706 --> 01:32:02,020 please. And the last slide from the team is the list of our sponsors. Huge thanks 787 01:32:02,020 --> 01:32:08,210 to all of them. It won't be possible to create such a huge event and such loads of 788 01:32:08,210 --> 01:32:15,490 infrastructure without them. And that's everything we have. 789 01:32:15,490 --> 01:32:25,707 ysf: Amazing. Thank you for all you've done. Truly incredible, and showing 790 01:32:25,707 --> 01:32:30,980 everything to the public. So I promised that there will be a kind of behind the 791 01:32:30,980 --> 01:32:37,130 scenes look of this infrastructure talk or review. And I really have nothing to do 792 01:32:37,130 --> 01:32:41,000 with it. Everything was done by completely different people. I'm only a Herald, 793 01:32:41,000 --> 01:32:47,040 somehow lost and tumbled into this stream. And so I'm just going to say switch to 794 01:32:47,040 --> 01:33:03,722 wherever. Show us the magic. Karlsruhe: Three hours ago, I got the 795 01:33:03,722 --> 01:33:10,592 call… Hello and welcome from the last part of the infrastructure review and greetings 796 01:33:10,592 --> 01:33:15,973 from Karlsruhe. So three hours ago, I got a call from lindworm and he asked me, how 797 01:33:15,973 --> 01:33:23,485 is it with this last talk we have? It may be a bit complicated. And he told me, OK, 798 01:33:23,485 --> 01:33:28,950 we have a speaker. I'm the Herald. Oh, that's always so. And then we realized, 799 01:33:28,950 --> 01:33:35,110 yeah, we don't have only one speaker, we have 24. And so that's why we called 800 01:33:35,110 --> 01:33:41,780 ChaosWest and built up an infrastructure which dampfkatze will explain you now in a 801 01:33:41,780 --> 01:33:48,200 short minute. I think so. dampfkatze: Thank you. Yes. Oh, I lost the 802 01:33:48,200 --> 01:33:57,710 sticker. OK, after we called ChaosWest, we came up with this monstrosity of the video 803 01:33:57,710 --> 01:34:08,777 cluster. And we start here. The teams streamed via OBS.Ninja onto three 804 01:34:08,777 --> 01:34:19,611 ChoasWest studios. They were brought together via RTMP on our Mix1 local 805 01:34:19,611 --> 01:34:31,030 studio, and then we pumped that into Mix2, which pumped it further to the VOC. The 806 01:34:31,030 --> 01:34:37,730 slides were brought in via another OBS.Ninja directly onto Mix2. They came 807 01:34:37,730 --> 01:34:43,746 from lindworm. Also, the closing you will see shortly hopefully will also come from 808 01:34:43,746 --> 01:34:52,790 there. And ysf and lindworm were directly connected via OBS.Ninja onto our Mix1 809 01:34:52,790 --> 01:35:02,709 computer. And Mix2 also has the studio camera you're watching right now. And for 810 01:35:02,709 --> 01:35:09,767 the background communication, we had a Mumble connected with our audio matrix. 811 01:35:09,767 --> 01:35:17,750 And lindworm, ysf, and the teams, and we in the studio locally could all talk 812 01:35:17,750 --> 01:35:24,164 together. And now back to the closing with… No, to the Herald News Show, I 813 01:35:24,164 --> 01:35:32,790 think. lindworm will introduce it to you. lindworm is live. 814 01:35:32,790 --> 01:35:51,640 lindworm: Is ysf still there? Or do you come with me? So it will take a second or 815 01:35:51,640 --> 01:36:02,412 billions of years. So thank you very much for this review. It was as chaotic as the 816 01:36:02,412 --> 01:36:04,800 Congress. 817 01:36:04,800 --> 01:36:16,850 *postroll music* 818 01:36:16,850 --> 01:36:27,780 Subtitles created by c3subtitles.de in the year 2021. Join, and help us!