avatarLarry | Peng Yang

Summary

The provided web content is a comprehensive guide on managing multiple Python versions and virtual environments using pyenv and pyenv-virtualenv on a MacBook Pro with macOS Monterey.

Abstract

The article serves as a detailed tutorial for Python developers looking to efficiently manage different Python versions and virtual environments on their macOS systems. It outlines the step-by-step process of installing pyenv via Homebrew to handle multiple Python versions, and it introduces pyenv-virtualenv for creating and managing project-specific virtual environments. The author emphasizes the importance of understanding the existing Python setup before making changes and provides instructions for uninstalling Python versions and related tools if needed. The guide aims to streamline the setup of development environments, ensuring that dependencies are isolated and manageable across various projects.

Opinions

  • The author believes that using pyenv and pyenv-virtualenv is a superior method for managing Python environments compared to other approaches like using brew or the official installers.
  • It is suggested that the ability to switch between Python versions and virtual environments with pyenv enhances flexibility and simplifies dependency management.
  • The author expresses that storing virtual environment data outside of the project directory (as done with pyenv+virtualenv) is preferable to the traditional approach of including it in the project (as done with virtualenv alone).
  • The article conveys the opinion that automating the activation and deactivation of virtual environments with pyenv-virtualenv is a significant advantage in development workflows.
  • The guide implies that a clean and well-managed Python environment is crucial for avoiding conflicts and ensuring reproducibility across development setups.

A step-by-step guide of Python versions and virtualenv management with pyenv

Photo by AltumCode on Unsplash

Why I wrote this blog?

Do you have experience configuring the Python development environment while referencing multiple browser tabs by switching them back and force to just get it to work? If you get lucky, you may end up with a “working” environment within 10 mins, but some may spend more than 30 mins if not an hour on trial and error. But does everyone understand how exactly the versions and virtual environments really work under the neath? It might be even worse if you have to do the same thing when you switch to a new development environment, like on a new machine.

In this article, I’m going to share the step-by-step practice on what and how I configured the different Python versions and virtual environments and also with the conclusion that you can just apply it to your configuration.

Environment

  1. MacBook Pro 2022 M2 with macOS Monterey 12.6
  2. Shell: Z Shell (zsh)
  3. Xcode is installed
  4. Python versions installed through this blog: 3.8.13, 3.9.13, 3.10.6. Don't worry, you can install whatever you want after reading this blog.

Before you start

Before even starting installing or configuring anything, you should at least understand what is on your machine already, here is the checklist for you.

# check installed python versions. 3.9.6 is system python3 (I believe it is)
$ python -V
zsh: command not found: python
$ python3 -V
Python 3.9.6
$ which python3
/usr/bin/python3
$ ls -lh /usr/bin/python3
-rwxr-xr-x  76 root  wheel   163K Aug 24 17:59 /usr/bin/python3
$ pip -V
zsh: command not found: pip
$ pip3 -V
pip 21.2.4 from /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages/pip (python 3.9)
$ which pyenv
pyenv not found
$ which virtualenv
virtualenv not found

Seeing the similar results above on your machine is a good sign as you will just need to follow the rest to make everything work. If not, you may want to uninstall them to keep them clean and simple before we proceed to the next step. Please see the last section of this blog to find how to uninstall Python, pyenv, virtualenv etc.

Installing and using multiple python versions

One of our goals is to keep the number of python packages as little as possible, so let’s only install the versions that we really want to have on the machine. There are at least 3 approaches:

  1. brew command
  2. pyenv command
  3. download from python.org and install it

Based on my experience, you wouldn’t go wrong if we stick with option 2 (pyenv CLI) if we are going to use virtual environments. So, let’s install pyenv first.

$ brew install pyenv
# check pyenv. the output might be /opt/homebrew/bin/pyenv depending on your or MacOS, after I reinstalled brew, I got /opt/homebrew/bin/pyenv instead of /usr/local/bin/pyenv
$ which pyenv 
/usr/local/bin/pyenv
$ pyenv -v
pyenv 2.3.4
$ pyenv versions
* system (set by /Users/pengyang/.pyenv/version)
# add the line to your profile, otherwise your system doesn't recoginize the python you installed with `pyenv` below.
$ echo 'eval "$(pyenv init -)"' >> ~/.zprofile
$ source ~/.zprofile

Then, let’s install some python versions.

# Check what versions are available
$ pyenv install --list | grep -e 3.8. -e 3.9. -e 3.10.
 3.8.9
 ...
 3.8.13
 3.9.9
 ...
 3.9.13
 ...
 3.10.4
 ...
 3.10.6
# Let's install the latest versions for 3.8.x, 3.9.x and 3.10.x
$ pyenv install 3.8.13
$ pyenv install 3.9.13
$ pyenv install 3.10.6
$ ls -lh /Users/pengyang/.pyenv/versions/
total 0
drwxr-xr-x  7 pengyang  staff   224B Oct  2 12:20 3.10.6
drwxr-xr-x  7 pengyang  staff   224B Oct  2 12:02 3.8.13
drwxr-xr-x  6 pengyang  staff   192B Oct  2 10:46 3.9.13

If the latest versions don’t show with pyenv install --list, try pyenv install <a version doesn’t show up> and checking the prompt, and then do either of the following based on the prompt.

$ cd /Users/pengyang/.pyenv/plugins/python-build/../.. && git pull && cd -
$ brew update && brew upgrade pyenv

Note that you may fail to install them if you don’t have xcode installed, if so, just install xcode from App Store or the CommondLineTools with xcode-select --install and the python versions installed here can be only recognized by pyenv.

Now, let’s check what we have done so far.

$ pyenv versions
* system (set by /Users/pengyang/.pyenv/version)
  3.8.13
  3.9.13
  3.10.6
# You still see system python as we haven't set what version to use with pyenv
$ python3 -V
Python 3.9.6
$ pip3 -V
pip 21.2.4 from /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages/pip (python 3.9)
# And python and pip commands are not available yet
$ python -V
pyenv: python: command not found
$ pip -V
pyenv: pip: command not found

We have installed our python versions, let’s make them available for us. There are two ways to set the python versions that we want to use for our projects.

  1. Set global python version with pyenv global x.x.x that applies to the whole machine.
  2. Set local python version with pyenv local x.x.x that is applied to the current directory (and it has higher precedence than the global one)

Let’s set a global one first.

$ pyenv global 3.9.13
$ cat ~/.pyenv/version # control global python version
3.9.13
$ pyenv versions
  system
  3.8.13
* 3.9.13 (set by /Users/pengyang/.pyenv/version)
  3.10.6
# We also notice pip and pip3, python and python3 share the same python version we specified.
$ which python; which python3; python -V; python3 -V
/Users/pengyang/.pyenv/shims/python
/Users/pengyang/.pyenv/shims/python3
Python 3.9.13
Python 3.9.13
$ which pip; which pip3; pip -V; pip3 -V
/Users/pengyang/.pyenv/shims/pip
/Users/pengyang/.pyenv/shims/pip3
pip 22.0.4 from /Users/pengyang/.pyenv/versions/3.9.13/lib/python3.9/site-packages/pip (python 3.9)
pip 22.0.4 from /Users/pengyang/.pyenv/versions/3.9.13/lib/python3.9/site-packages/pip (python 3.9)
# Now you can use python or python3 command and the version will be the same everywhere on your machine
# Also you can use pip command to install dependencies.

In the case that you may want to use a different version for a project, you can do the following by setting the local python version.

$ mkdir local-prj && cd local-prj
$ python -V
Python 3.9.13
$ pyenv local 3.10.6
 ✔ 3:12:16 [pengyang@MacBook-Pro] $ local-prj $
$ cat .python-version
3.10.6
 ✔ 3:12:19 [pengyang@MacBook-Pro] $ local-prj $
$ python -V
Python 3.10.6
 ✔ 3:12:27 [pengyang@MacBook-Pro] $ local-prj $
$ pip -V
pip 22.2.1 from /Users/pengyang/.pyenv/versions/3.10.6/lib/python3.10/site-packages/pip (python 3.10)
# unsurprisingly, in local-prj directory, python version 3.10.6 is used.

Working with multiple virtual environments

Up until now, we have demonstrated how to keep multiple python versions and switch them with pyenv commands which is quite flexible compared with the case if we install with brew or installers. Although we can now install the python dependencies for our project with pip install <package> either in a local or global python environment, the dependencies are installed in the same folder for that python version. Let’s say we have local-prj1 and local-prj2 and they both use the local python version 3.10.6. It may be troublesome if we need different dependencies versions for different projects as the dependencies are located in the same folder.

$ cd local_prj2
$ pyenv local 3.10.6
$ python -c "import site;print(site.getsitepackages())"
['/Users/pengyang/.pyenv/versions/3.10.6/lib/python3.10/site-packages']
$ cd ../local_prj1
$ pyenv local 3.10.6
$ python -c "import site;print(site.getsitepackages())"
['/Users/pengyang/.pyenv/versions/3.10.6/lib/python3.10/site-packages']

Wouldn’t it be great if not only we can use different python versions for different projects, but also different dependencies (including dependencies versions) for projects? Virtual environments are here to save our lifives. Like there are several ways to install python, there are also several approaches to working with virtual environments.

  1. Simple approach: use virtualenv installed with pip command. A simple case is as below. (alternatively, you can use venv command, see here)
$ pip -V
pip 22.0.4 from /Users/pengyang/.pyenv/versions/3.9.13/lib/python3.9/site-packages/pip (python 3.9) # global is used
$ pip install virtualenv
$ mkdir local_prj
$ cd local_prj; pyenv local 3.10.6
$ pip -V
pip 22.2.1 from /Users/pengyang/.pyenv/versions/3.10.6/lib/python3.10/site-packages/pip (python 3.10)
$ virtualenv --version
virtualenv 20.16.5 from /Users/pengyang/.pyenv/versions/3.10.6/lib/python3.10/site-packages/virtualenv/__init__.py

$ virtualenv local_env # createlocal_env dir with local python version 3.10.6
$ source local_env/bin/activate # activate virtual env
# Install packages for local_env
(local_env) $ pip install <package_name>
# exit the env
(local_env) $ deactivate
# delete the env
$ rm -rf local_env

2. A better solution: pyenv + virtualenv

The major differences between the two approaches are:

  • In approach 1, the virtual environment is created in the project folder (which means we need to exclude it from the version control system) while the latter one keeps them in /Users/pengyang/.pyenv/versions/<python_version>/envs/<env_name>.
  • With approach 2, we can activate and deactivate the virtual environment automatically when we go into/out of the project directories which is very helpful in most if not all cases.
  • In approach 2, we specify the python version when creating the virtual environment while the python version depends on the pyenv global/local setting in approach 1.

Let’s see how approach 2 works by starting with installing the command we need.

$ brew install pyenv-virtualenv
To enable auto-activation add to your profile:
  if which pyenv-virtualenv-init > /dev/null; then eval "$(pyenv virtualenv-init -)"; fi
# And then do below to enbale auto activation/deactivation
$ echo 'if which pyenv-virtualenv-init > /dev/null; then eval "$(pyenv virtualenv-init -)"; fi' >> ~/.zprofile
$ source ~/.zprofile

And then, let’s create two virtual environments for 2 projects where the same python version is used, let’s reuse the local_prj1 and local_prj2.

# First create 2 virtual envs
$ cd ~
$ pyenv virtualenv 3.10.6 prj1_env
des=/Users/pengyang/.pyenv/versions/3.10.6/envs/prj1_env
$ pyenv virtualenv 3.10.6 prj2_env
des=/Users/pengyang/.pyenv/versions/3.10.6/envs/prj2_env
$ pyenv versions
  system
  3.8.13
* 3.9.13 (set by /Users/pengyang/.pyenv/version)
  3.10.6
  3.10.6/envs/prj1_env
  3.10.6/envs/prj2_env
  prj1_env
  prj2_env
# Now we activate virtual env prj1_env for local_prj1
$ cd local_prj1
$ pyenv activate prj1_env
pyenv-virtualenv: prompt changing will be removed from future release. configure `export PYENV_VIRTUALENV_DISABLE_PROMPT=1' to simulate the behavior.
(prj1_env)  ✔ 3:53:27 [pengyang@MacBook-Pro] $ local_prj1 $
(prj1_env) $ python -c "import site;print(site.getsitepackages())"
['/Users/pengyang/.pyenv/versions/prj1_env/lib/python3.10/site-packages']
# deactivate the env
$ source deactivate
# let's do it in a smarter way for local_prj2
$ cd ../local_prj2
$ python -V
Python 3.10.6 # local python version
# Notice that the env is activatelly and deactivated automatically when we switch the locations!
$ pyenv local prj2_env
(prj2_env)  ✔ 3:58:06 [pengyang@MacBook-Pro] $ local_prj2 $ # activated
(prj2_env) $ cd ..
 ✔ 3:58:41 [pengyang@MacBook-Pro] $ ~ $ # deactivated 
$ cd -
~/local_prj2
(prj2_env)  ✔ 3:58:50 [pengyang@MacBook-Pro] $ local_prj2 $ # activated
(prj2_env) $ python -c "import site;print(site.getsitepackages())"
['/Users/pengyang/.pyenv/versions/prj2_env/lib/python3.10/site-packages']
(prj2_env) $ pip -V
pip 22.2.2 from /Users/pengyang/.pyenv/versions/3.10.6/envs/prj2_env/lib/python3.10/site-packages/pip (python 3.10)
# Now you can install dependecies separately for local_prj1 and local_prj2 with pip install <package_name> and the dependencies are stored in seperate folders.

Everything works like a charm! Even though we had set local python versions for two projects, and the virtual environments used the same python version, it doesn’t have to be so —the project will use what python is in the virtual environment (specified when creating the env) that is attached to it.

To delete the virtual environments, we can simply run pyenv virtualenv-delete <env_name>.

How to uninstall Python, pyenv, and virtualenv

This section provides the information which you may need if your initial environment is a dirty one. Here is some quick guidance on how to uninstall some of them.

Uninstall python which is installed by brew

# Uninstall python if they are previous installed by brew, pyenv or installers.
$ ls -l /usr/local/bin | grep -i Python3
lrwxr-xr-x  1 pengyang  admin         42 Oct  2 07:43 python3 -> ../Cellar/[email protected]/3.10.6_2/bin/python3
lrwxr-xr-x  1 pengyang  admin         49 Oct  2 07:43 python3-config -> ../Cellar/[email protected]/3.10.6_2/bin/python3-config
lrwxr-xr-x  1 pengyang  admin         45 Oct  2 07:43 python3.10 -> ../Cellar/[email protected]/3.10.6_2/bin/python3.10
lrwxr-xr-x  1 pengyang  admin         52 Oct  2 07:43 python3.10-config -> ../Cellar/[email protected]/3.10.6_2/bin/python3.10-config
lrwxr-xr-x  1 pengyang  admin         42 Jul 13  2020 python3.8 -> ../Cellar/[email protected]/3.8.3_2/bin/python3.8
lrwxr-xr-x  1 pengyang  admin         49 Jul 13  2020 python3.8-config -> ../Cellar/[email protected]/3.8.3_2/bin/python3.8-config\
# The the word `Celler` in the paths indicate they are probably installed with brew command. So check if brew recogonizes them.
$ brew list | grep -i python
# If they are shown with a `check` mark, then we can uninstall python3.8 and python3.10 with below. Follow what the promot asks if you face any errors.
$ brew uninstall python@3.8
$ brew uninstall python@3.10
# Then you will proabably see a verion other than what you have just uninstalled and the output of `ls -l /usr/local/bin | grep -i Python3` is like to be empty
$ python3 -V
Python 3.9.6
$ ls -l /usr/local/bin | grep -i Python3

Uninstall python which is installed by official installers

# find the Python3.7 folder in `Application` and delete it, and then do below
$ sudo rm -rf  /Library/Frameworks/Python.framework
# find the symlinks that are pointing to the version we just deleted and delete them
$ ls -l /usr/local/bin | grep Python
lrwxr-xr-x  1 root       wheel        66 Sep 10 02:08 2to3 -> ../../../Library/Frameworks/Python.framework/Versions/3.7/bin/2to3
lrwxr-xr-x  1 root       wheel        70 Sep 10 02:08 2to3-3.7 -> ../../../Library/Frameworks/Python.framework/Versions/3.7/bin/2to3-3.7
lrwxrwxr-x  1 root       admin        78 Sep 10 02:08 easy_install-3.7 -> ../../../Library/Frameworks/Python.framework/Versions/3.7/bin/easy_install-3.7
lrwxr-xr-x  1 root       wheel        67 Sep 10 02:08 idle3 -> ../../../Library/Frameworks/Python.framework/Versions/3.7/bin/idle3
lrwxr-xr-x  1 root       wheel        69 Sep 10 02:08 idle3.7 -> ../../../Library/Frameworks/Python.framework/Versions/3.7/bin/idle3.7
lrwxrwxr-x  1 root       admin        66 Sep 10 02:08 pip3 -> ../../../Library/Frameworks/Python.framework/Versions/3.7/bin/pip3
lrwxrwxr-x  1 root       admin        68 Sep 10 02:08 pip3.7 -> ../../../Library/Frameworks/Python.framework/Versions/3.7/bin/pip3.7
lrwxr-xr-x  1 root       wheel        68 Sep 10 02:08 pydoc3 -> ../../../Library/Frameworks/Python.framework/Versions/3.7/bin/pydoc3
lrwxr-xr-x  1 root       wheel        70 Sep 10 02:08 pydoc3.7 -> ../../../Library/Frameworks/Python.framework/Versions/3.7/bin/pydoc3.7
lrwxr-xr-x  1 root       wheel        69 Sep 10 02:08 python3 -> ../../../Library/Frameworks/Python.framework/Versions/3.7/bin/python3
lrwxr-xr-x  1 root       wheel        76 Sep 10 02:08 python3-config -> ../../../Library/Frameworks/Python.framework/Versions/3.7/bin/python3-config
lrwxr-xr-x  1 root       wheel        71 Sep 10 02:08 python3.7 -> ../../../Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7
lrwxr-xr-x  1 root       wheel        78 Sep 10 02:08 python3.7-config -> ../../../Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7-config
lrwxr-xr-x  1 root       wheel        72 Sep 10 02:08 python3.7m -> ../../../Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7m
lrwxr-xr-x  1 root       wheel        79 Sep 10 02:08 python3.7m-config -> ../../../Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7m-config
lrwxr-xr-x  1 root       wheel        68 Sep 10 02:08 pyvenv -> ../../../Library/Frameworks/Python.framework/Versions/3.7/bin/pyvenv
lrwxr-xr-x  1 root       wheel        72 Sep 10 02:08 pyvenv-3.7 -> ../../../Library/Frameworks/Python.framework/Versions/3.7/bin/pyvenv-3.7
# delete the following 4 lines from profile (~/.zprofile)
# Setting PATH for Python 3.7
# The original version is saved in .zprofile.pysave
PATH=”/Library/Frameworks/Python.framework/Versions/3.7/bin:${PATH}”
export PATH
# delete the file
$ rm ~/.zprofile.pysave
# confirm which python is used now. The output varies based on your env
$ python3 -V
$ which python3

Uninstall python which is installed by pyenv

$ pyenv versions
* 3.7.5 (set by xxx/xxx)
  3.8.5
  3.9.2
# let's uninstall python 3.9.2
$ pyenv uninstall 3.9.2
$ pyenv versions
* 3.7.5 (set by xxx/xxx)
  3.8.5

Uninstall pyenv and pyenv-virtualenv

# remove pyenv related lines in profile and then
$ rm -rf $(pyenv root)
$ brew uninstall pyenv
$ brew uninstall pyenv-virtualenv

Uninstall virtualenv that is installed with pip

$ pip uninstall virtualenv

Conclusion

  1. Difference python versions can be installed via brew, pyenv or installers
  2. Python versions can be switched easily with pyenv. Python used in pip3 is based on the pyenv global/local settings. If pyenv is set up properly, python and python3 have the same output, pip and pip3 have the same output.
  3. pyenv-virtualenv is used to manage virtual environments
  4. We compared 2 ways of using virtual environments where the pip+virtualenv approach stores env data in project dir while pyenv+virtualenv approach stores them in ~/.pyenv. and approach 2 is more flexible although it requires more configurations.

References

  1. file:///private/var/folders/z1/s4s6nn_x0x31ld6whvtvw8400000gn/T/178.html
  2. https://sourabhbajaj.com/mac-setup/Python/virtualenv.html
  3. https://jordanthomasg.medium.com/python-development-on-macos-with-pyenv-virtualenv-ec583b92934c
  4. https://www.ianmaddaus.com/post/manage-multiple-versions-python-mac/#uninstall-python
  5. https://stackoverflow.com/questions/51797189/how-to-uninstall-pyenvinstalled-by-homebrew-on-mac
  6. https://github.com/pyenv/pyenv-virtualenv
Python
Pyenv
Virtualenv
Recommended from ReadMedium
avatarAnthony M
Installing Python

Installing Python on Windows

2 min read
avatarEmad Dehnavi
How to install Poetry in MacOS

Poetry is a tool for dependency management and packaging in Python. While there are many ways to install Poetry in your mac and you can…

2 min read
avatarJonathan Mondaut
How ChatGPT Turned Me into a Hacker

Discover how ChatGPT helped me become a hacker, from gathering resources to tackling CTF challenges, all with the power of AI.

4 min read
avatarTed James
[SOLVED] Conda env vs venv / pyenv / virtualenv / etc

2 min read
avatarAbhay Parashar
17 Mindblowing Python Automation Scripts I Use Everyday

Scripts That Increased My Productivity and Performance

14 min read
avatarLEARN TO CODE
14 Python Automation Scripts to Simplify Your Daily Tasks

Python is a versatile programming language that can be used to automate a wide range of tasks, saving time and reducing errors. Whether…

3 min read