A Conversation with Linus Torvalds at Open Source Leadership Summit 2017: transcript

Key points

• The kernel is organized at every level so people can work independently, which lets many changes happen in parallel even though individual pieces move slowly.
• Quality does not come from Torvalds reading every patch; he stopped doing that about 20 years ago because he became a bottleneck, and relies on a chain of trust between maintainers.
• Trust belongs to the person rather than the company, and newcomers build it by sending many small patches.
• He is surprised by how far Git has gone, credits Junio Hamano as maintainer for ten years, and says GitHub was key to its spread.
• Open source works like evolution rather than intelligent design, with competing projects trying ideas in parallel.
• The roughly two-and-a-half-month release cycle, in place for about ten years, replaced painful multi-year releases, and he says process problems are worse than code problems.
• He says innovation hype is meaningless and that 99% of the work is getting things done.

Chapters

1. 0:00Welcome and opening banter
2. 1:23The pace of Linux kernel development
3. 4:33Quality, trust and maintainers
4. 9:41Git, GitHub and distributed development
5. 14:19Requirements, evolution and competing projects
6. 16:26Release cycles and the value of good process
7. 20:5225 years on: the work matters more than innovation

Full transcript

0:00I would like to start off by welcoming someone who I think most of you are all familiar with to the stage. We've got a great lineup of speakers today, but the first one, I think, needs very little introduction. He created Linux. He created this thing called Git. He created this great diving application that many of you may be unaware of. Please welcome Linus Torvalds. Welcome. Thank you. And as usual, I don't do talks.

0:32I hate preparing talks, especially when I don't know what people are expecting or interested in. So blame Jim if the questions are not interesting. All right. Well, yeah, well, let me just, this is pretty much a puff piece interview here. We're going to, this is not a hard hitting journalism. But you know, it's funny because you come to a lot of our events and I know that you like scuba diving. so you go diving around a lot of events i saw you brought your skis uh to this event no no i was carrying skis but they were not actually mine oh okay so you're not no so usually i only go to places where i can do fun stuff But then occasionally it's Valentine's Day and I go to places where my wife can do fun stuff.

1:23I noticed that she's up on the mountain somewhere now. Exactly. Because someone pulled me aside and said, scuba diving, downhill skiing, Jim, what are you doing here with Lena Storvalds? And what I told them was... Apparently, my mic is not working. But apparently it's not skiing or scuba diving that are dangerous to you. it's the constant fear of being hit by a bus that's what people always ask me about yeah I don't know especially in the US it makes no sense at all to me it makes more sense in on the European circuit but well we're gonna talk about what happens if you get hit by bus in a minute here but you know one of the things that I think shocks a lot of people particularly people who are new to open source or who are working on projects is the pace of Linux development. So we asked Greg for some statistics and what he roughly described was that 10,000 lines are added to 8,000 lines are subtracted from and 1,800 odd lines are modified in Linux

2:27every single day. That's sort of the pace of development, which is amazing. And, you know, there's a lot of developers in here who just think about that pace of development and just are shocked. I mean, my first question to you is, you've been doing this a long time. How do you come up with a system. That allows that kind of development pace to to continue. Well, so the kernel is kind of special. It is the biggest project. It's by far the one that also has the biggest amount of code written for it. Just the pace thing. And part of part of it is, we've been doing this for 25 years. And one of the constant issues we've had is people stepping on each other's toes.

3:15So for all of that history, what we've done is organize the code, organize the flow of code, organize our just maintainership so that the main pain point, which is people just disagreeing about a piece of code, basically goes away. So the kernel is literally organized at every level so that people can work independently. And it is slightly, I mean, what happens is people see that there's a huge influx of patches and there's a, we have, like, the statistics really are staggering, but then when you look at any particular one single piece of code, the development is not necessarily fast at all.

4:03It can take... I mean, some patches go in just in a day or two, but it can take months, it can take a year for big reorganizations to happen, so each individual piece may not be, it's not a breakneck speed kind of thing, it's just that we've been able to really modularize the code and modularize the whole development model to the point where we can just do a lot in parallel, and that's what then allows this high flow in the end.

4:33but you're not maybe you are you're not reading all this code yourself like how do you how do you develop the ability to you know when you do your final release to to be sure that that's that's code that's a decent quality and it's uh it's so it's a social project i mean it's about technology and the technology is what makes people able to agree on issues because technology tends to have very They're not black and white, but at least there's usually a fairly clear right and wrong.

5:08So you can have people agree on the end result, but at the same time the thing that just me ensuring quality, it's not because I read through all the patches. I used to, and it became a huge problem because I ended up being, and this is 20 years ago, I became a bottleneck. So I don't do that. that word. It's the social network, the trust and finding the people who can do the work, and I trust that they may not do the same code reading either, but they have people they trust, and we have a very strong network, and that's why we can have a thousand people involved in every release, because of those thousand people, a lot of them end up being these levels of trust where there's checking at every level not of the code but of

6:09well a code at the lowest levels but as you move up the maintainership chain and the chain is often three four or five people you always have that level of trust and the level of just knowing people i've many of the people i work with i've literally worked with for over 25 years uh there are new people coming in also but but there's there's this whole human like potential there that we use very extensively to make sure the process works and it's taken 25 years to get to where we are i mean right on a process level too yeah and and you know what's What's interesting is that We did a survey and I think 95% of the developers who work on Linux, particularly in the core project, are all professional developers who get paid by various companies, many of which are represented here. But the trust that you've built is with the individuals in that project. And that's taken a lot of time.

7:14Yeah, and it's very noticeable, too, when people move between companies, the trust does not stay with the company, the trust stays with the person. And obviously it's not like people stay around forever. People come and go, and that's, I think, one of the reasons people sometimes think it's hard to get into the kernel is because you think you can just come in and shoot off a patch and go away, and that's not how it works. What really takes time is to shoot off a lot of small patches until you reach the point where the maintainers trust you.

7:49And at that point, you become something more than just the guy who sends patches. You become part of the whole network of trust. Yeah, yeah. It's interesting. You know, we have some projects. Cloud Foundry is an interesting project in that they use a pair of programming technique, which is an interesting technique out of Pivotal Labs where developers are programming together. Same code, two screens, in order to kind of accelerate that trust and empathy that, you know, in the kernel takes a long time.

8:20I mean, the kernel is kind of odd. And what we do does not necessarily always make sense in other settings. Just the fact that we, for example, cannot have people sit next to each other just means that we have to work differently than you necessarily would do in some other company. Or maybe there's a lot of projects that are much more localized to them than the kernel. Even when they're open source and even when they use email or the internet to communicate.

8:51Most of the time, a lot of projects tend to be inside companies or maybe inside universities or maybe it's just that there's a lot of like there's the trust came from previously knowing the person so there's a lot of advantage to being close by and being part of this like inner group and in Linux we never had that and I attribute some of it to me being from Helsinki Finland which meant that most of the people by far that I could work with were all remote I mean I did not have the kind of local group around me that you find from Like some of the traditional Unix projects, like obviously the BSDs, which actually happened inside a particular university.

9:41Yeah. And you, so also Linux in order to, the project in order to handle this pace. I mean, I love the social construct, but you also needed special tools. And so you created Git to do that. And now we're in this, you know, what some people like to call this sort of post GitHub era, where, you know, yesterday, someone from GitHub was here and shouted out, that there are 50 million repositories on GitHub. And what advice do you give to GitHub projects? First, are you surprised about where Git has gone?

10:14And then what advice would you give to projects that are on Git in order to help them build trust relationships and to do this kind of work at scale? So on the surprise part, I'm really surprised by where Git has gone. Because obviously Git was written to be solved the problems we had in the kernel community. And nobody else, or it looked for a long time like nobody else cared about the whole distributed environment and distributed development.

10:44And in fact for the maybe three, four first years of Git. One of the biggest complaints about Git was how it was so different and everybody had to learn something new and people hated it. I mean, people who came, who weren't inside the kernel and were trying to learn it after learning CVS or something like that. And then I'm really surprised by how it switched and apparently really people started seeing the whole distributed development model.

11:20and having companies like GitHub just taking advantage of the distributed development model and making it so easy to then host Git repositories, which was never true if you don't have distributed. The problem with centralized models tended to be that you had exactly one point that was the central one, which means that it's really hard to trust a new company or any hosting company, right? Because if you're a hosting company and you host the one centralized repository, you are so critical that it's really hard to even get there.

12:02And being distributed just meant that now it's much easier to be a hosting company and give services and value add. What I'm trying to say is it really took a long time and then everything like within a year it went from having a lot of people complaining about how it was different to just growing like mad and it took me by surprise and I'm very pleased obviously because CVS has been around for way too long and there were a lot of other source control management.

12:35Sounds like it still is out here. Well, I mean, it exists, and I'm not trying to say CVS was evil 40 years ago. I'm more saying that time had passed it by, and no other source control system seemed to ever be able to really take over, and Git seems to have done so. And that's obviously a source of pride. But every time I talk about Git, I want to make it very, very clear that I was the maintainer for Git for six months.

13:11And that's 10 years ago, right? So I came up with the design, I came up with the basics, and then Junia Hamano has been such an incredible maintainer for 10 years that that's always something to keep in mind that all credit goes to him apart from that initial spark. GitHub, no credit? No, I do think that the fact that... I mean, without GitHub, there's other companies too, but GitHub is clearly the 800-pound gorilla.

13:43Without that kind of infrastructure, Git wouldn't have gone anywhere. And I think the fact that GitHub existed and there were a few projects that decided, hey, this model just works very well. Ruby, OJS. All the Ruby projects switched like suddenly, and I think that was one of the things that drove Git to be more than just the kernel people and the geeky people who really cared. No, and GitHub has done an amazing job of, you know, and that's why people call it the sort of post-GitHub world of just the phenomenal development that's going on.

14:19So you got the tools, you got the trust relationships. What about, one question that I have, and then that comes up often for a lot of our other projects is for the kernel it seemed like for the first long period of time you were fast following an incumbent technology and requirements sort of flowed from that but that has not been the case for a long time how in addition to how do you process so much code how do you understand what requirements are out there what features and functionality you want to integrate in well i I mean, that's the real advantage of open source, I think, is that if you want to do Something new. Something that really is pushing the envelope. The only way you can really, I mean, nobody can really say this is the way to do it. So what you need to do is just a lot of trial and error. And I often compare software development to biological processes where it really is evolution. I mean, it is not intelligent design. I'm there in the middle of the thing and I can tell you it is.

15:34absolutely not intelligent design, whatever you want to think. You have to have, I mean, you have a lot of people who have their ideas of where things need to go. And then you need to have a marketplace where you can try them out. And that's where open source allows all these different groups to just go off, do their own thing, and then come back and say, hey, look, I did this and it actually worked. and that's very hard to do in in any other environment because if you're inside a company you can't afford to waste those kinds of resources if you're inside a company you can't say okay let's just explore this whole space and put ten different teams on it and see what works that does not work I mean I I suspect a lot of companies have tried it, but they probably only try it once.

16:26Right. But in open source, that's literally what we do, particularly in the kernel. We have competing projects. We have a lot of code that never, ever gets merged. Because somebody went off, they cloned my tree, they did their thing, they could never get it to work. Or they could kind of get it to work, but it only worked for their very particular platform. which is fine too don't get me wrong it's these niche things that never go anywhere else uh and that never get merged back because there's no point and but you did at one point the kernel decided to do a long-term stable release and then have intermittent releases in between what brought that decision about so we've gone through so many different models and And they've all had huge problems until we then figured out about 10 years ago the model that we're on now, which seems to work really well.

17:28The traditional model is you take three years to make a new release and you aim for a certain I mean, there's usually many different features, but there's a few flagship features that you know, that's what you're working on. And the reason it takes three years is it takes that long to get big new features done. And in the meanwhile, so you have this development tree that all the geeks are excited about and that does the new stuff, but it's also slightly unstable.

18:03So, most of the traditional users tend to stay back. And that kind of worked, but it meant we did that for probably five, six years. And it caused horrible problems when you then did the switchover because so much had changed. So few people had tested the development kernels because they were considered to be unstable. So only the most excited people or the most forward-thinking people, and there's very few of those, were testing development kernel in order to make sure that they worked for their workload.

18:43So whenever we then finished a stable release, we were always in huge pain. And it just, it does not scale. It does not work. It's, it was, it was a disaster. But it took us a long time to figure out that it was so bad. Yeah. And then 10 years ago, and nobody believed this would work. Literally. Everybody was like laughing in the room. And at the Colonel Summit we've said, I basically said, I want to have an eight week release schedule. Two months, that's it, right?

19:14and everybody knew I was like pushing for for something unrealistically short so that maybe you could get it down to six months from three years right but but you aim for two months in order to hit six months and now we've actually done the the release cycle is usually two and a half months and we've done it for ten years and it's lovely it really works it has avoided all the problems we've used to have it solved so many things we didn't even know needed to be solved so it's almost boring how well our process works and i say almost boring because it's so much relaxing more relaxing when process works yeah all the really stressful times for me personally have been about process they've never been about code when code doesn't work that can actually be exciting right that's that can be fun that can be the okay you hit your head on a wall for a couple of weeks and it can be very frustrating but then when you solve it the

20:17The feeling of accomplishment is huge, right? I'm hoping people in the audience know that, right? Code, technical problems can be challenging and frustrating, but they're good things to have. Process problems are a pain in the ass. You never, ever want to have process problems. And that's why I'm so happy that our process has worked for the last 10 years and hopefully will continue to work because that's when when people start getting really angry at each other, when the process doesn't work.

20:52Well, last question. You know, 10 years is a long time, but you've been releasing the Linux kernel for 25 years. You know, it's just an amazing amount of fortitude for anyone to do the same thing for 25 years. I understand you took a six-month break, it sounds like, to do this Git thing. but it was actually more just two months like a very short break for a minor minor effort but the last question is sort of you know you got almost 900 organizations that participate in all the different Linux foundation projects here that are creating code at in almost every aspect of computing and these aren't just you know a fast following president these are the core main projects for the automotive sector, the telecom sector.

21:47What, for you, was there a moment where you thought, aha, this is huge, this is much bigger than maybe what you thought when you started? Was there any particular thing that you? I've answered that before. There hasn't really been anything like that. There's been a couple of moments that took me by surprise, but the big moment was actually when it went from past being personal. Like, and that was within six months of just releasing it. It was, I I don't even know the people working on sending me patches, like know them personally.

22:21I mean, obviously now I know them by email, but there's a ton of people I've never met. And just going past that initial personal phase where the project was just for me, I do want to kind of mention the whole 25-year thing. I'm a huge believer in just the 99% sweat, 1%, 99% perspiration, 1% inspiration thing.

22:52Any technology project, the innovation that this... Industry talks about so much is bullshit. Innovation. Anybody can innovate. Don't do this big think different. Don't do this big innovation thing. Screw that. It's meaningless. Like 99% of it is get the work done, right? And there are, I actually, that's my least favorite part of the technology, like news cycle, is this constant innovation and new ideas and this will revolutionize all that hype is is that's not where the real work is yeah the real work is in the details and I'm obviously one of those people who just I like to concentrate on one project I don't like flitting from one idea to another and you need the

23:56people who just flutter about and come up with ideas but they're not the really useful ones they're the ones who they end up being the ones ideas the people who then do the work so I hope you guys that's What anybody should take away from this talk is the people who actually do the work are the ones you should really listen to, and these days I don't actually do the work anymore. I merge other people's work, but I'd like conferences, too, to be less about the visionary innovation thing and more about the day-to-day, what are my problems and how do I want to solve them.

24:42well always inspiring and depressing simultaneously so I think what it's fair to say is you know real innovation happens when you kind of get over yourself and sit down and do the work which is fair enough we I think that's a sage advice so thank you for coming it's always a pleasure Lisa Sorvalds

Source and licence

Recording: File:Keynote, A Conversation with Linus Torvalds (Efi9wa4Fwds).webm by The Linux Foundation (2017-02-17), via Wikimedia Commons. Licence: CC BY 4.0.

The transcript is machine speech-to-text by the TidyTools Audio & video transcriber; the summary, key points and chapters were written by Claude from that transcript. Both may contain errors: check the recording before quoting it.