Under Discussion: Uncovering Project Python

After the first edition of dotPy was cancelled this March due to the COVID-19 outbreak, we got on a video conference call with scheduled speakers Carol Willing and Łukasz Langa, who are both very much part of the Python ecosystem. Willing is a member of the Python Steering Council as well as a Python Software Foundation fellow and core developer, while Langa is the current release manager of Python 3.8 and 3.9. Together, we talked about how Python governance entities are organized, how evolutions of the language are decided on, and what areas of the language they would like to see developed in the coming years.

What’s your current involvement in the Python ecosystem?

Carol Willing: I’ve been involved in the Python community for about eight years. Right now, I’m on the Python Steering Council and I’m also a core developer and a Python Software Foundation fellow, which means that I’ve made some sort of significant contribution to Python or the community. Last year I was really touched and honored to receive the Frank Willison Award from the Python Software Foundation for technical achievement and community building within Python. I’m also very much involved with Project Jupyter, which is best known for the Jupyter Notebook and allows you to do interactive computing and computational narratives that are widely used in the sciences, data sciences, and digital humanities. Python is something that’s near and dear to my heart, even though I actually really enjoy many languages. When I first started using it, what I really enjoyed about it was the syntax and readability of it. And then, as I got more involved with Python, I loved the community that was built around the language even more, partially because of the strong focus on education and helping others to create things using the language.

Łukasz Langa: I’m the current release manager of Python 3.8, which is the latest version of Python that is already stable and released—this is the one you should be using in production—as well as Python 3.9, which is now available for testing. Apart from that, I am the maintainer of a code formatter called Black, along with Carol, actually. I also worked for Facebook for more than five years. Now I work remotely from Poland for a company called EdgeDB on a relational database written in Python 3.8.

Why do you think Python is so popular currently?

CW: I definitely think its readability and the focus that the language has had on education are two big drivers of its popularity! Another is the fact that Python can be used in so many different applications in industry. Whether it’s system administration or embedded computing, it’s a very versatile language. One of the benefits of the versatility is there is a rich ecosystem of third-party libraries and packages, so you can pretty much get whatever you want done with many things that are already written and out there in the wild.

Another thing that makes Python highly successful is performance. And not performance from the standpoint of how fast it is, in terms of actually running on hardware, because there are other languages that are statically typed and compiled that may run faster. I mean that, in terms of getting projects done quickly from prototype to production, Python is excellent and one of the best at being able to do that quickly, efficiently, and reliably.

What are the roles of the existing governance entities around the Python project?

CW: The Python Software Foundation is typically what people first see when they become part of the Python community. They are the ones who maintain the Python.org website. Their mission is to protect and advance the intellectual property of Python, as well as grow its international community and reach as many users as possible in a variety of industry disciplines or particular areas. And in order to let people get new conferences started, run workshops, and really introduce Python into new global areas that, perhaps, the language hadn’t been present in 10 years ago, they raise funds through the PyCon conferences, sponsored donations, corporate donations. I was a board member on the Python Software Foundation, and some of what we did was deliver grants to people.

Another governance entity is the Python Steering Council that is responsible for the core development of the language itself from a technical standpoint, as well as the experience that core developers and contributors might have when adding to the language or trying to make sure that it is robust and bug free and secure, and things like that. It was created in 2018 in response to Guido van Rossum’s resignation as the benevolent dictator for life. He was on the inaugural steering council and this year has chosen to really follow some of his technical and mentorship passions. The steering council is currently made up of five people, who are elected by the core developers. We’re not necessarily telling people what to do, we’re helping them get the things done within development that they need to get done. This could be improving the ecosystem around how we manage things on GitHub, which is the Python Enhancement Proposal [PEP] process, which is what any major new feature would go through to determine whether or not it really belongs in the language, and if so, in what capacity. And then we also do things like steward the next releases in the language.

Those two entities meet once a week to work in partnership. For example, Ewa Jodlowska, who is the executive director of the Python Software Foundation, participates in the weekly meetings of the Python Steering Council.

LL: And the last entity is the core team. In theory there are a hundred of us in the core developer team. In reality, there are probably 35 people who are actually actively doing things to Python as we speak, and who are actually involved with changes to the code base or the next version. The team is organized around committing to the repository. There is very little in terms of structure inside the team, so the team is, to a large extent, self-organizing. That means that it is a big deal to become a core developer, because once you are a part of the team, you pretty much have privileges to do whatever you want with Python. Usually people steer towards responsibilities and areas of the code base that they feel they most need experience in and want to get involved with. You’re going to have people who are like, “Hey, I’m really into making Python work better with multicore systems,” so they’re going to be working on the interpreter itself. There are also people who are almost exclusively working on making the documentation work better and who would make sure that, whenever I release a new version of Python, when there’s a drop-down where you can select the version, it always steers you to the newest release.

CW: One thing I think is really important is that the vast majority of people who are core developers for Python are volunteers and are contributing to it in their own time. There’s a handful of people who may have some release time from their companies to contribute to Python, but that is the exception as opposed to the norm. As a consequence, you can’t tell people to do things, because they’re volunteers. You can encourage, you can show the merits of perhaps doing something. But at the end of the day, we are very reliant on our volunteers, and part of the steering council’s charter is to create an environment for volunteers to be successful and collaborate successfully.

Are there any companies involved in Python’s governance entities?

LL: We don’t really have any individual companies involved in Python. This is actually one of the biggest differences between Python and pretty much all other programming languages, because looking at any other programming language, you can see that either there is a single company controlling it, such as Apple with Swift or Google with Go, or there is a committee of companies controlling it, as is the case with JavaScript and C++.

This independence regarding companies is a blessing and a curse at the same time. The blessing is that there’s no particular company that can tell us what we should be doing. However, there is also a lack of long-term direction. When van Rossum was the benevolent dictator, he provided some of that because he was the person with a vision of where he wanted Python to go. With the steering council now, I hope we’re going to be pushed in areas where we should be going long-term. It would also be a massive blessing to have financial backing of a company or a bunch of them, because there are many ways to back Python, but they are often multi-step processes before the money reaches developers. And that, again, causes people to only work on stuff that is not only their hobby but also kind of easy and small.

CW: What the steering council is looking to do, for the better part of six months last year and for all of this year, is exactly what Łukasz is saying. How do we improve the ability to engage or allow engagement from these large companies that may have an interest in something that is foundational, so that they can run their software faster, better, more securely? So that they can invest in that process and help us move it forward while also letting us carry out something that has been very close to Python for a long time, which is the education of not just children or college students but everyone across the world? Python serves so many different communities—education, large companies that are running their business on top of Python, where performance is very important, data science, which has very different needs, as well as system administration and many different areas. That sometimes makes it difficult to say there is one path.

How do the different governance entities make sure they don’t break things for developers when evolving Python?

LL: Even if there is a bug, even if there is a clear thing that we know is wrong, there is often still a painful and long process to determine whether it’s worth fixing. If the bug is new, it will probably not be a problem, we should just fix it, it’s fine. But if the bug has been there for 10 years, well it’s probably better to just leave it, since there are for sure workarounds for that bug already and plenty of code in production. And you have to keep in mind that there is between 10 to 100 times more of that coding that will never see the light of day—gigantic corporations power plenty of their work with Python. So if you fix a bug, you will break their programs by fixing a problem. I’m thinking here of the example of urlparse(). There was a bug that was actually fixed and released, and then it turned out that there were people who were relying on the incorrect behavior. We had to revert that change for the latest version of Python so that it behaves as planned. So the solution is to fix it by introducing a new function of things that matches what we had always wanted the old one to do. That’s often what we actually have to do.

CW: The last thing I would want to do is break something related to healthcare! Especially in these sorts of times [with the COVID-19 pandemic], fixing something might break their stuff. That’s where the third-party libraries and things outside the standard library become very important, because they give people additional ways to extend the language without us having to cause an impact on the history and the folks who are using older releases.

What is the process for deciding whether a new feature should be implemented?

CW: If it’s anything sizable or anything that would have an impact on potentially backwards compatibility, it would go through the PEP process. This typically comes from some of the mailing lists, like Python ideas, or work that core development has done or other people have.

LL: Yeah, so there are plenty of venues where we discuss those kinds of early-stage proposals, and the Python ideas mailing list is one of them. All the features requests need to be documented through a form. And sometimes what happens is that features that we never thought of doing finally get accepted once someone writes the PEP form. The fact that we are actually using a narrative form, where we can actually go ahead and read through, means people change their minds all the time. The biggest example was the introduction of assignment expressions, which was part of Python 3.8, which we initially planned to reject. I’m actually using them a lot right now!

And who makes the final decision to accept or reject a feature once it has been discussed?

CW: Historically, van Rossum, as the benevolent dictator for life, was the one who would make those difficult decisions, or he would appoint a delegate to make that decision. With the change in the governance that we had after his retirement from benevolent dictator for life, it is now the steering council or a delegate of the steering council who would have to make that hard decision. However, the council tends to try to build consensus as opposed to applying the firm hand of “Yes, we’re going to do this.” At the end of the day, we want what’s best for the language. We want what’s going to make the language successful and sustainable. So it’s better to have many voices, even if it’s painful during the process, to actually get to a good resolution in the end.

LL: It’s not every time that everybody ends up being happy with a decision, but that’s kind of a function of having a diverse set of people involved. Of course, we can talk about diversity later—there are plenty of ways in which we are not diverse enough. However, there are plenty of different opinions of course, there are plenty of groups of interest. So, data scientists for example, who are very important to us, actually have a lot of input there. Having those different factions that pull in different directions makes sure we’re not over-optimizing in a particular area.

What are the main areas of the language that are going to evolve in the years to come?

CW: From my perspective, there are several things that I would like to see in the future. First and foremost, I want to make sure that Python continues the commitment to education and sustains the usability of the language. I would be very sad to see it become a language that’s hard to read, that’s hard to get started using. Then there’s the runtime performance of Python, which a lot of work has been done to optimize, but we’re not a compiled and statically typed language at the core. My hope is that it will be part of Python’s future, because I think the large web companies, the sciences, data science, all benefit from that. Even the delivery of online education, which given current events, has become even more important, relies on that. The third thing would be that you want to keep the people who are involved in the creation of Python sustained, fulfilled, and rewarded for the wonderful things that they’re bringing to the world.

LL: I have three answers here, actually. First, I would like to talk about some things that are part of Python 3.9, and maybe 3.10 later. They’re not very revolutionary, but are nevertheless interesting. One of the things I’m personally involved in is making typing more comfortable to use, typing in the sense of using static typing annotations, so that a type checker like Mypy can help us discover problems with our code before that code is even run. We’ve had this for a while, since Python 3.5, with the typing module and there have been a few iterations on additional features over the past five years. For instance, I’ve been involved in the future import annotations that already makes it easier for you to hold references, for example. So if you are declaring a type of argument, and that type’s class is only defined later, that’s fine with Python now. It didn’t use to be. There’s another PEP—PEP 585—which makes it easier to use static typing with Python and which is now part of Python 3.9.

Another thing that I work on, but have not published anything yet, is making exception groups possible in Python. In the async I/O world, where you have a single thread and that single thread is doing many things, when something goes wrong, it might not be just one thing that went wrong, it might be multiple things that went wrong at the same time. And you would like to know about all the things that went wrong, and frankly, in Python, that isn’t possible. We would like it to be possible, but we would also like it to compose with the existing exception. It is a very tricky thing to make right. It’s a brave and unprecedented thing in computer science, so we’ll see how that works. It will probably be in Python 3.10.

And for the longer term, what I would like to see is better performance, which is going to be achieved in two ways, I guess. Either we can have a new form of Python, in terms of maybe Python for the browser—kind of Python-like but not really Python. The other natural way to speed it up is what PyPy did, which is introduce just-in-time compilation. PyPy on its own is a separate project with a separate core team and separate goals, but I do deeply believe that without just a plain compilation as part of CPython, we’re missing out! We could make Python much faster while just making it observe what the users’ program is doing and compiling machine code to do exactly the same thing.

CW: I like all of those, Łukasz! Especially the async I/O exceptions. Anybody who programs with asynchronous code would probably be, in theory, happy to see something like that. But it’s a difficult problem. I think with all those improvements to come, we want to keep that excitement high. That passion and desire to make the language better foster those ideas that will benefit everyone in the long term but are maybe complicated to get done. And it’s through partnerships and other things that we will get them done, as opposed to people feeling as if their ideas are not being listened to or valued. What we want is looking toward the future to be a positive experience, not a limiting one!

This interview has been edited for space and clarity.

This article is part of Behind the Code, the media for developers, by developers. Discover more articles and videos by visiting Behind the Code!

Want to contribute? Get published!

Follow us on Twitter to stay tuned!

Illustration by Catherine Pearson