Python is Ushering in Real Multi-Threading

The Global Interpreter Lock(GIL) can be removed. From then on, Python will no longer be what people call pseudo-multithreading.

“GIL in Python will be no more. Huge win for AI ecosystem..” PyTorch core maintainer Dmytro Dzhulgakov said with emotion.

What is the GIL? The full name of GIL is Global Interpreter Lock, which is not unique to Python, but a concept introduced when implementing CPython (Python interpreter). We can understand GIL as a mutex, which is used to protect objects in Python and prevent multiple threads from executing Python bytecode at the same time, thereby ensuring thread safety.

However, GIL has a disadvantage, that is, only one thread can execute on one CPU at a time, and multiple threads cannot be mapped to multiple CPUs, so Python cannot achieve true multi-thread concurrency, thereby reducing execution efficiency.

Now, the Python team has officially accepted the proposal to remove the GIL and made it an optional mode, which is good for developers.

This contribution was made by a software engineer at Meta named Sam Gross, and it took him over four years to complete the project.

After hearing the news, everyone applauded. Yann LeCun, one of the three giants of deep learning, sent a message of congratulations: Without the GIL, Python code can now freely execute multi-threading.

No more GIL in Python.

Removing the GIL from Python presents a future where the language’s capabilities can grow exponentially.

For the details, let’s see below.

CPython core developer Thomas Wouters writes about GIL-free Python in detail and looks to the future.

Thanks a lot for everyone’s feedback on the no-GIL proposal, and overall positive support. The Steering Committee intends to accept the no-GIL proposal and share specific details with you below.

The basic assumptions are:

• In the long term (roughly 5+ years), no-GIL builds should be the only builds;
• We want to be very careful about backward compatibility. We don’t want another Python 3 situation where all 3rd party code changes needed to accommodate no-GIL builds should only apply to with-GIL builds (although backward compatibility issues with older Python versions still have to be addressed). This doesn’t work with Python 4. We are still considering the requirements for ABI compatibility and other details of these two builds, and the impact on backward compatibility;
• Would like to see community support before we commit to fully moving to no-GIL. We can’t just change the defaults, we want the community to figure out what they’re doing to support us. Our core development team needs to gain experience with the new build mode and everything related. We need to sort out thread safety in existing code and need to figure out new C APIs and Python APIs. We also need to convey these insights to others in the Python community and ensure that the changes we want to make and the changes we want others to make are desirable;
• At any point before our default no-GIL setting, we would like to be able to change our minds if it proved to be too disruptive for too little benefit. This also means that we roll back all our work, so no-GIL-specific code should be somewhat identifiable until we decide to make no-GIL the default.

Currently, we see the road ahead in three phases:

• In the short term, we will make no-GIL builds an experimental build mode, probably in 3.13 (and possibly 3.14). It’s experimental because our core development team while supporting this build mode, doesn’t expect the entire community to support it. We need time to figure out what we want to do, at least in terms of API design, packaging, and distribution, so that we can get support from the community. We also discourage distributors from releasing experimental no-GIL builds as the default interpreter.
• In the medium term, after we are confident that we have enough community support to make production use of no-GIL feasible, we will support no-GIL builds, but not by default, but at a certain target date or in a certain Python version using It becomes the default way. The exact timing will depend on many factors, such as how compatible the API changes end up being, how much work the community thinks they still need to do, etc. We expect this to take at least one to two years. Once we announce support, expect some distributors to start shipping no-GIL by default.
• Long term, we want no-GIL to be the default and remove all traces of the GIL (but without unnecessarily breaking backward compatibility). We don’t want to wait too long, after all, the existence of two commonly used build modes at the same time will cause a great burden on the community (such as the need to double-test resources and debug scenarios). But we can’t be too hasty. We think this process will take five years.

Of course, throughout the process, our entire development team will need to evaluate progress in real-time and make adjustments to the timeline.

What are your thoughts on GIL being optional?

In Plain English 🚀

Thank you for being a part of the In Plain English community! Before you go:

• Be sure to clap and follow the writer ️👏️️
• Follow us: X | LinkedIn | YouTube | Discord | Newsletter
• Visit our other platforms: Stackademic | CoFeed | Venture | Cubed
• More content at PlainEnglish.io