I have been using Python for a number of years now and one of the biggest frustrations I have when teaching people how to use it is managing packages and versions of Python itself. Most operating systems still come with Python 2.7 by default which is used to run software on the system. Where people start running into trouble is when they start using that version of Python for their own work. Since that version of python is being used by the OS, we really don’t want to mess with it as changing/updating its modules could cause problems with the way OS scripts are running. It can also be a bit of a hassle when we want to try out a new package, and later find that we no longer need it. Removing a package and it’s dependencies without screwing up other packages can be a bit of a nightmare. To help with these types of issues, python has introduced virtual environments.
Python virtual environments will allow you to run any version of Python and install any packages you like. When you are done or if you screw the environment up, you can delete the environment and create a new one. You can have multiple environments installed at the same time, each with its own version of Python and different sets of modules. The environments are not virtual machines, so no CPU resources are being used and you don’t have to remember to turn then on or off. This post will go through the steps of installing a package called virtualenvwrapper - a very powerful and easy to use virtual environment manager for Python.
Let’s dig in. These instructions are for Mac OSX but will be almost identical on any version of linux, just substitute brew with apt-get, yum, emerge, pacman or whatever your system uses to manage packages. Unlike most linux distros, Mac OSX does not come with a package manager. So if you don’t already have it, install HomeBrew (brew) now. You need to be on OSX 10.13 (Sierra) or higher and XCode which can be found in the AppStore (which takes a log time to download - but you’ll need anyway if you are doing dev work). The HomeBrew website is here https://brew.sh and brew can be installed by running a simple curl command:
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
[... lots of messages here ...]
With brew installed, we can now install the latest version of Python:
brew install python3
[ ... lots more messages ... ]
brew is downloading the packages/code for Python 3 and installing it in /usr/local on your system. When it’s done, you should be able to find python in /usr/local/bin/
ls /usr/local/bin/python3
/usr/local/bin/python3
/usr/local/bin/python3 -V
Python 3.7.7
pip -V
/usr/local/bin/pip 20.0.2 from /Library/Python/2.7/site-packages/pip-20.0.2-py2.7.egg/pip (python 2.7)
/usr/local/bin/pip3 -V
pip 20.0.2 from /usr/local/lib/python3.7/site-packages/pip (python 3.7) The next step in to install the virtual environment manager using pip3:
pip3 install virtualenvwrapperAh, that was pretty easy right. We need to do a bit of configuration now. First we want to create a directory in our home directory (or wherever you want it to be) that will hold all of the environments. the WORKON_HOME environment variable is used by virtualenvwrappers to locate all of the environments, so we will set this first, then make the directory and then initialize the environments:
export WORKON_HOME=~/.virtualenvs
mkdir $WORKON_HOME
export PYTHONPATH=$PYTHONPATH:/usr/local/lib/python3.7/site-packages
source /usr/local/bin/virtualenvwrapper.sh
Now we can add this to the .bash_profile (.bashrc on linux) file so that it works when we start a new Terminal window. You can also add /usr/local/bin to your PATH if you don’t already have it:
# ~/.bash_profile: executed by bash(1) for login shells.
# see /usr/share/doc/bash/examples/startup-files for examples.
# the files are located in the bash-doc package.
[... other start up code ...]
# include /usr/local/bin in your path if you don't already have it
export PATH=$PATH:/usr/local/bin
# Set up virtualenvwrappers for Python
export WORKON_HOME=~/PythonEnvs
source /usr/local/bin/virtualenvwrappers.sh
Finally, create a new environment:
mkvirtualenv test1
Using base prefix '/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7'
New python executable in /Users/wyllie/PythonEnvs/test1/bin/python3.7
Also creating executable in /Users/wyllie/PythonEnvs/test1/bin/python
Installing setuptools, pip, wheel...
done.
virtualenvwrapper.user_scripts creating /Users/wyllie/PythonEnvs/test1/bin/predeactivate
virtualenvwrapper.user_scripts creating /Users/wyllie/PythonEnvs/test1/bin/postdeactivate
virtualenvwrapper.user_scripts creating /Users/wyllie/PythonEnvs/test1/bin/preactivate
virtualenvwrapper.user_scripts creating /Users/wyllie/PythonEnvs/test1/bin/postactivate
virtualenvwrapper.user_scripts creating /Users/wyllie/PythonEnvs/test1/bin/get_env_details
Your command line prompt will change to look something like (test1) wyllie@dilex:~ $ The (test1) will be the name of the environment you are now using. You can check this by looking at the versions of python and pip:
python -V
Python 3.7.7
which python
/Users/wyllie/PythonEnvs/test1/bin/python
pip -V
pip 20.1 from /Users/wyllie/PythonEnvs/test1/lib/python3.7/site-packages/pip (python 3.7)
which pip
/Users/wyllie/PythonEnvs/test1/bin/pip
A couple of things to notice here. You can see that python and pip are coming from a path in my home directory, the cool part is that the default versions point to version Python v3.7.7. I don’t have to type python3 or pip3 or any of that nonsense.
Let’s install some packages now. How about numpy and scikit:
pip install numpy scikit-learn
>>Collecting numpy
>> Using cached numpy-1.18.3-cp37-cp37m-macosx_10_9_x86_64.whl (15.1 MB)
>>Collecting scikit-learn
>> Using cached scikit_learn-0.22.2.post1-cp37-cp37m-macosx_10_9_x86_64.whl (7.1 MB)
>>Collecting scipy>=0.17.0
>> Using cached scipy-1.4.1-cp37-cp37m-macosx_10_6_intel.whl (28.4 MB)
>>Collecting joblib>=0.11
>> Using cached joblib-0.14.1-py2.py3-none-any.whl (294 kB)
>>Installing collected packages: numpy, scipy, joblib, scikit-learn
>>Successfully installed joblib-0.14.1 numpy-1.18.3 scikit-learn-0.22.2.post1 scipy-1.4.1Your output may look slightly different - I already had these packages downloaded so they were retrieved from cached versions. You can see where the packages were installed by looking in $WORKON_HOME/test1/lib/python3.7/site-packages/ or by using the lssitepackages command:
ls -l $WORKON_HOME/test1/lib/python3.7/site-packages/
>>total 8
>>drwxr-xr-x 3 wyllie staff 96 Apr 29 10:22 __pycache__/
>>-rw-r--r-- 1 wyllie staff 126 Apr 29 10:22 easy_install.py
>>drwxr-xr-x 29 wyllie staff 928 Apr 29 10:22 joblib/
>>drwxr-xr-x 8 wyllie staff 256 Apr 29 10:22 joblib-0.14.1.dist-info/
>>drwxr-xr-x 30 wyllie staff 960 Apr 29 10:22 numpy/
>>drwxr-xr-x 10 wyllie staff 320 Apr 29 10:22 numpy-1.18.3.dist-info/
>>drwxr-xr-x 7 wyllie staff 224 Apr 29 10:22 pip/
>>drwxr-xr-x 9 wyllie staff 288 Apr 29 10:22 pip-20.1.dist-info/
>>drwxr-xr-x 8 wyllie staff 256 Apr 29 10:22 pkg_resources/
>>drwxr-xr-x 8 wyllie staff 256 Apr 29 10:22 scikit_learn-0.22.2.post1.dist-info/
>>drwxr-xr-x 37 wyllie staff 1184 Apr 29 10:22 scipy/
>>drwxr-xr-x 9 wyllie staff 288 Apr 29 10:22 scipy-1.4.1.dist-info/
>>drwxr-xr-x 44 wyllie staff 1408 Apr 29 10:22 setuptools/
>>drwxr-xr-x 11 wyllie staff 352 Apr 29 10:22 setuptools-46.1.3.dist-info/
>>drwxr-xr-x 52 wyllie staff 1664 Apr 29 10:22 sklearn/
>>drwxr-xr-x 14 wyllie staff 448 Apr 29 10:22 wheel/
>>drwxr-xr-x 9 wyllie staff 288 Apr 29 10:22 wheel-0.34.2.dist-info/
lssitepackages
>>__pycache__/ scikit_learn-0.22.2.post1.dist-info/
>>easy_install.py scipy/
>>joblib/ scipy-1.4.1.dist-info/
>>joblib-0.14.1.dist-info/ setuptools/
>>numpy/ setuptools-46.1.3.dist-info/
>>numpy-1.18.3.dist-info/ sklearn/
>>pip/ wheel/
>>pip-20.1.dist-info/ wheel-0.34.2.dist-info/
>>pkg_resources/
You can create as many virtual environments as you like, so you can have different environments for different sets of packages. To see all of your environments, simply type workon
workon
>>demo
>>datasci
>>junky
>>test1
workon junkyecho $VIRTUAL_ENV >>/Users/wyllie/PythonEnvs/junky
The deactivate command will put you back on the systems Python
deactivate
python -V
>>Python 2.7.16
You can remove an environment with the rmvirtualenv command. You can’t remove the environment you are currently using though, so you may have to deactivate first:
deactivatermvirtualenv test1
>>Removing test1...
…and that’s about it.
It’s handy to get into the habit of tracking which modules you are using with at requirements.txt file. This file just list all of the modules you are going to use on a particular project. This makes it very easy to rebuild your environment and track what’s in it. This is also necessary if you want to do any CI/CD type work where you will have to build your project in a virtual environment.
As an example, say I want to build a virtual environment and load up numpy and scikit-learn again. I’ll create a requirement.txt file with the two packages listed, build the environment and then load the packages:
cat requirements.txt
>>numpy
>>scikit-learn
mkvirtualenv datasci
>>[ ... lots of output ... ]
pip install -r requirements.txt
>>[ ... lots of outout ... ]
python -c "import sklearn; sklearn.show_versions()"
>>
>>System:
>> python: 3.7.7 (default, Mar 10 2020, 15:43:33) [Clang 11.0.0 (clang-1100.0.33.17)]
>>executable: /Users/wyllie/.virtualenvs/datasci/bin/python
>> machine: Darwin-19.4.0-x86_64-i386-64bit
>>
>>Python dependencies:
>> pip: 20.1
>>setuptools: 46.1.3
>> sklearn: 0.22.2.post1
>> numpy: 1.18.3
>> scipy: 1.4.1
>> Cython: None
>> pandas: None
>>matplotlib: None
>> joblib: 0.14.1
>>
>>Built with OpenMP: True
You can do a lot of requirement.txt files including locking down version numbers in cases where different modules version do not play nice with each other.