Converting conda/pip environments into Docker Images

Using Coiled as a Docker Image pipeline

tl;dr

Building Docker images is a necessary but challenging part of most data scientists’ workflows. One way to make this process easier is by hooking Coiled up to your own Docker registry (like DockerHub) to create a pipeline that converts your conda or pip environments into Docker images that can be used elsewhere. This post shows you how to do that using the coiled.create_software_environment() command and by setting the container registry backend in the "Account" settings your Coiled cloud dashboard.

You can also check out the video below for a step-by-step tutorial:

Disclaimer: I work at Coiled as a Data Science Evangelist Intern. Coiled is founded by Matthew Rocklin, the initial author of Dask, an open-source Python library for distributed computing.

Why you might need Docker images

Many data scientists need to convert conda and/or pip environments into Docker images, for collaboration across teams or to move local working environments to run on the cloud. Most people achieve this manually using workflows that can get increasingly elaborate (for example, have a look at this Medium post.) While this technically works, it’s not always accessible to everyone. As a novice Data Scientist, I know I found the experience complex and difficult to master at first.

How Coiled can make this easier for you

As a cloud-based service designed for data scientists, Coiled needed to be able to convert conda and pip environments into Docker images to run correctly. The de>coiled.create_software_environment()command converts a environment.yml (conda) or a requirements.txt (pip) file into a Docker image, which is then distributed to all Dask workers to ensure consistent dependencies across your Coiled cluster.

Below are two code snippets: one for conda and one for pip:

# using conda .yml files
import coiled
 
coiled.create_software_environment( 
    name='my-env', 
    conda='<path/to/environment.yml>' 
)

# using pip .txt files
import coiled

coiled.create_software_environment( 
    name='my-env', 
    pip='<path/to/requirements.txt>' 
)

In the spirit of community and open-source development, Coiled engineers did not tuck this functionality away but have made it a general-purpose tool. This means you can hook up Coiled to your own Docker registry (like DockerHub) to create a conda/pip-to-Docker build service. You can then use the Docker images created using Coiled anywhere you like.

How to Connect Coiled to your Docker Container Registry

By default, Coiled stores the software environments you create in the container registry of the cloud service you are running on: either AWS, GCP, or Azure. You can change this setting within the “Account” tab of your Coiled Cloud dashboard to have your software environments saved as Docker images to your Docker Hub.

Any registry that supports the Docker Registry API V2 should work. Read more details in the “Backends” page of our docs.

Please note: using registries other than Docker Hub is an experimental feature under active development. Please reach out if you would like to discuss your use case at [email protected] or via the Coiled Community Slack channel.

And I’ll sneak in a bonus hint here: creating software environments does not actually require you to spin up a Cluster. This means you can use your conda/pip-to-Docker pipeline without burning any of your Coiled credits.

The Value of Inventing Nothing

This is what I love about the Python ecosystem. Functionalities that are integral to a product can be opened up for broader, public use. It follows the counterintuitive Principle of Minimum Creativity that Matt Rocklin has often proclaimed about the open-source library Dask: “Invent nothing.”

This mantra may seem counterintuitive but actually holds a simple lesson: instead of trying to reinvent the wheel, create products that integrate with the tools that people are already using, thereby adding an extra brick to the rich ecosystem that already exists.

I hope you found this post helpful! Follow me on Twitter for daily data science content.

Or came say hi at my blog: CrunchCrunchHuman:

Originally published at https://coiled.io on August 23, 2021.