Avoid the Snake Pit of Python Package Management With Poetry

Using Poetry to build private and public Python libraries

Python is a programming language that brings joy to a lot of developers. Unfortunately, building and using libraries in Python has always been a struggle. Multiple tools exist that try to help (to name a few: setuptools, pip, virtualenv, pipenv, tox, and conda) but each one has its own quirks and limitations.

In this article, we look at Poetry, a relatively new package management tool that tries to bring the same joy of programming in Python, to building and using Python libraries.

Poetry in practice

The steps below show you how to build a library with Poetry, how to use that library from another Python project, and how to distribute the library for public reuse on PyPi.

Step 1. Installing Poetry

The Poetry website contains extensive documentation on how to install it on different operating systems.

I’ve installed Poetry on my MacBook through Homebrew:

$ brew install poetry
$ poetry --version
Poetry version 1.1.11

Step 2. Creating a small Python library from scratch

Let’s write a tiny library using Poetry. This library will be called PyInitials and its single purpose is to return initials for given full names, for example, “Guido van Rossum” => “GvR”.

First, initialize a new library project with Poetry:

$ poetry new pyinitials
Created package pyinitials in pyinitials

Next, create a new virtual environment and install the initial set of dependencies:

$ cd pyinitials
[pyinitials] $ poetry install
Creating virtualenv pyinitials-Q6tcgwA_-py3.9 in /Users/rob/Library/Caches/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies... (1.0s)

Writing lock file

Package operations: 8 installs, 0 updates, 0 removals

• Installing pyparsing (2.4.7)
  • Installing attrs (21.2.0)
  • Installing more-itertools (8.10.0)
  • Installing packaging (21.0)
  • Installing pluggy (0.13.1)
  • Installing py (1.10.0)
  • Installing wcwidth (0.2.5)
  • Installing pytest (5.4.3)

Installing the current project: pyinitials (0.1.0)

After that, initialize Git and commit all generated files:

[pyinitials] $ git init
[pyinitials] $ git checkout -b main
[pyinitials] $ git add *
[pyinitials] $ git commit -m "Initial commit"

Finally, create a new repository called pyinitials on GitHub, add this repository as a remote to your local Git repository, and push all changes:

[pyinitials] $ git remote add origin git@github.com:<YOUR_GITHUB_USERNAME>/pyinitials.git
[pyinitials] $ git branch -M main
[pyinitials] $ git push -u origin main

Step 3. Add some code and tests

Open the generated file test/test_pyinitials.py and add a first unit-test:

def test_initials():
    from pyinitials.initials import initials
    assert initials('Guide van Rossum') == 'GvR'

Next, run the unit-tests (using Poetry):

[pyinitials] $ poetry run pytest
===================== test session starts ======================
platform darwin -- Python 3.9.7, pytest-5.4.3, py-1.10.0, pluggy-0.13.1
rootdir: /Users/rob/Desktop/pyinitials
collected 2 items

tests/test_pyinitials.py .F                               [100%]
================= 1 failed, 1 passed in 0.06s ==================

This will fail (not a big surprise) due to the import of a missing function called initials.

To fix the test add the following code to a new file: pyinitials/initials.py:

def initials(fullname: str) -> str:
    return 'GvR'

Next, run the unit-tests again to check they all pass:

poetry run pytest
===================== test session starts ======================
platform darwin -- Python 3.9.7, pytest-5.4.3, py-1.10.0, pluggy-0.13.1
rootdir: /Users/rob/Desktop/pyinitials
collected 2 items

tests/test_pyinitials.py ..                               [100%]
====================== 2 passed in 0.02s =======================

Step 4. Setting up Continuous Integration

To have the unit-tests run on every code push you can setup a simple workflow in GitHub Actions (there are specific Actions for Poetry available that you can use.)

Create a new file in you repository called ci.yml in the directory .github/workflows with the following content:

name: 'CI'

on: [push, pull_request]

jobs:
  ci:
    runs-on: ubuntu-latest
    steps:
    - name: 'Checkout'
      uses: actions/checkout@v2
    - name: 'Set up Python'
      uses: actions/setup-python@v2
      with:
        python-version: 3.9
    - name: 'Set up Poetry'
      uses: snok/install-poetry@v1
    - name: 'Install dependencies'
      run: poetry install --no-interaction --no-root
    - name: 'Run unit-tests'
      run: poetry run pytest

Add all new files to Git and push your changes to GitHub. On the “Actions” tab of your repository in GitHub you should now see the first run of the CI pipeline:

Step 5. Using the library in a private organization

To use the PyInitials library in a private organization (or your own public python projects) you can add it as a Git dependency. Using GitHub for private libraries is a great alternative to setting up a full fledged package repository.

To demonstrate how this works you first need to create a new Python project, again with Poetry of course:

$ poetry new mycoolproject
$ cd mycoolproject
[mycoolproject] $ poetry install

Then, add the dependency to the main branch of the GitHub pyinitials repository:

[mycoolproject] $ poetry add git+https://github.com/<YOUR_GITHUB_USERNAME>/pyinitials.git#main

The # suffix can be used to select branches, tags or even specific commit hashes, in this case the main branch is selected. If your repositories are private use git+ssh:// links and make sure everyone on the team (and your CI/CD pipelines) can access the repository using private keys.

To check the library was included successfully, start a shell inside your Poetry virtual environment and make a call to it:

[mycoolproject] $ poetry shell
Spawning shell within /../mycoolproject-2UmXNoUn-py3.9
$ python
Python 3.9.7 (default, Sep  3 2021, 12:37:55)
>>> from pyinitials import initials
>>> initials('Guido van Rossum')
'GvR'

Step 6. Publishing to PyPi

Poetry has two commands that make publishing to the central Python package repository a breeze: build and publish

(Before publishing a library to the open source community, make sure your code and documentation follow the best practices from the Python Packaging User Guide ✨)

First you need to build the distribution builds:

[pyinitials] $ poetry build
Building pyinitials (1.0.0)
  - Building sdist
  - Built pyinitials-1.0.0.tar.gz
  - Building wheel
  - Built pyinitials-1.0.0-py3-none-any.whl

The directory dist now contains a source distribution (sdist) file called pyinitials-1.0.0.tar.gz and a pre-built distribution (wheel) file called pyinitials-1.0.0-py3-none-any.whl.

Next you can publish both distribution builds to the Python package repository:

[pyinitials] poetry publish
Publishing pyinitials (1.0.0) to PyPI
 - Uploading pyinitials-1.0.0-py3-none-any.whl 100%
 - Uploading pyinitials-1.0.0.tar.gz 100%

And, voilà, that’s it, shipped! 🚀

Conclusion

Poetry is a tool that definitely brings the joy back in Python package management. Creating private and public libraries becomes easy with the concise and consistent set of Poetry commands.

The commands shown above is exactly how I publish and maintain the PyInitials package on PyPi (that can also generate initials for names other than Guido van Rossum).

Give Poetry a try in your next Python project, it has a gentle learning curve.

Happy coding!

If you enjoyed 😃 this post…

you might also like other things I write. Subscribe here to receive an update when I post something new.