Summary

This article explains the different types of Python package distributions, focusing on the advantages of Wheels over Source Distributions and Eggs.

Abstract

The article begins by introducing the concept of Python package distributions and their importance in Python development. It explains the two primary distribution types: Source Distributions and Built Distributions. Source Distributions are simpler, containing all the Python source code, data files, and a setup.py file. However, they pose a security risk due to the post-install setup.py build step. Built Distributions, on the other hand, pre-interpret or "build" the package, eliminating the need for a post-install build step. The article then delves into the different types of Built Distributions, namely Wheels and Eggs, highlighting the advantages of Wheels over Eggs and Source Distributions.

Opinions

The author suggests that Wheels are a more secure and efficient option than Source Distributions and Eggs.
The author advises against using Eggs due to their older format and the fact that they have been replaced by Wheels.
The author recommends publishing both Source Distributions and Wheels for a package, allowing consumers the option to use the wheel for convenience and the ability to fall back to Source Distributions if necessary.
The author emphasizes the portability and compatibility of Wheels, as they do not contain .pyc files and have a richer naming convention that communicates compatibility.
The author highlights that pip will always choose a wheel file over a Source Distribution file by default.
The author suggests that Source Distributions are more difficult to maintain across a large number of different supported platforms and versions.
The author is the co-founder of Ochrona, a company that focuses on improving Python security by managing open-source dependencies in the software development life cycle.

Understanding Python Package Distribution Types

What are Wheels, Eggs, and Source Distributions?

Based on photo by Erol Ahmed on Unsplash

If you’ve done much Python development you’re probably familiar with importing dependencies using pip, or even easy_install, if you’ve been at this for awhile. Whether you were aware of it or not, these dependencies likely came from the public Python Package Index (PyPI) or perhaps an internal mirror of the PyPi repository that is hosted by your company.

What you may not have been aware of is how these dependencies are actually packaged, delivered, and installed, and the differences between the different distribution types available for Python.

The Primary Distribution Types

There are two primary distribution types in use today, Built Distributions and Source Distributions.

Python Source Distributions

Python source distributions are the simpler of the two. A source distribution, or more commonly sdist, is a distribution that contains all of the python source code (i.e. .py files), any data files that the library requires, and a setup.py file which describes to the setuptools module how your python code should be packaged.

You can build an sdist distributable by running python setup.py sdist. By default this will generate the distribution in the form of a tarball (i.e. a gzipped tar file).

Upon installation of an sdist, the setup.py file is executed on the host, ensuring that that package can be installed correctly. While this can a be convenient way for package authors to ensure that a package and any dependencies or extensions are available, it also introduces a security risk and can be difficult to maintain across a large number of different supported platforms and versions.

Python Built Distribution

A built distribution, also sometimes referred to as a bdist, is slightly more complex in that it first pre-interprets or “builds” the package and critically cuts out the post-install setup.py build step that is necessary when using sdist.

It should also be noted here that bdist is often misunderstood to stand for binary distribution. While binary distributions are a type of bdist, not all bdists are considered binary distributions.

There are a few kinds of bdist files, but at a high level first there were python Eggs, and now there are python Wheels.

Wheels and Eggs are both built distribution packaging formats that are meant to remove the need to build or compile a package, but instead provide a distributable that can be unpacked and used immediately. Eggs are an older format introduced in 2004 and have since been replaced by Wheels, which were introduced in 2012. Don’t use Eggs.

There is a common misconception that wheels only contain compiled python, commonly seen as .pyc files. Wheels actually cannot contain .pyc files, while Eggs did contain .pyc files, though Wheels can contain other pre-compiled resources and at install time pip will automatically generate .pyc files that are compatible with the correct python interpreter — ensuring compatibility.

Python Eggs and wheels are actually distributed as renamed .zip files, with Eggs using the .egg extension and Wheels using .whl. Wheels utilize a standard naming structure which helps communicate their compatibility.

{dist}-{version}|-{build}|-{python}-{abi}-{platform}.whl

Advantages of Wheels over sdist

Offer faster installation of Pure Python packages and those with C extensions.
sdist requires the execution of arbitrary code to build and install a package which is slower, more difficult to maintain, and a security risk.
Wheels are much smaller than sdist files.
By default pip will always choose a wheel file over sdist file.

Advantages of Wheels over Eggs

Wheels do not contain .pyc files and therefore are more portable than Eggs.
Wheels have a richer naming convention which allows them to more easily communicate compatibility.
Eggs did not have a standard or originate from a PEP, and therefore could contain incompatible implementations.

What to use?

While the advantages of Wheels seems obvious, often times the best option is to publish both an sdist and a wheel if you’re able to. Assuming your package is pure python, this should be fairly easy to accomplish by running python setup.py sdist bdist_wheel. This way you allow consumers of your package the option to use the wheel for convenience, but the ability to fall back to sdist if they have a specific requirement to do so.

Hey, I’m Andrew Scott, co-founder of Ochrona. Ochrona focuses on improving python security by helping manage your open-source dependencies at every step in your SDLC.