Python Virtual Environments

It is often useful to have one or more Python environments where you can experiment with different combinations of packages without affecting your main installation. Python supports this through virtual environments. The virtual environment is a copy of an existing version of Python with the option to inherit existing packages. A virtual environment is also useful when you need to work on a shared system and do not have permission to install packages as you will be able to install them in the virtual environment.

Installing packages using pip and virtual environments

This guide discusses how to install packages using pip and a virtual environment manager: either venv for Python 3 or virtualenv for Python 2. These are the lowest-level tools for managing Python packages in a Python environment and are recommended if higher-level tools do not suit your needs.

The term package refers to a Distribution Package which is different from a Import Package that which is used to import modules in your Python source code.

Installing pip

You can use the Linux distribution-provided versions of pip, or, install pip yourself.
It’s recommended to use the system pip to bootstrap a user installation of pip :

python3 -m pip install --user --upgrade pip

Notice that you use python -m to update pip. The -m switch tells Python to run a module as an executable.Python loads the module in memory and allows the package to be removed while it is being used. You can run packages as if they were scripts if the package provides a top-level script __main__.py.

Check the pip:

jerry@jerry-Latitude-E6410:~/$ python -m pip --version
pip 19.0.3 from /home/jerry/anaconda3/lib/python3.7/site-packages/pip (python 3.7)

jerry@jerry-Latitude-E6410:~/$ python3 -m pip --version
pip 19.2.3 from /usr/local/lib/python3.6/dist-packages/pip (python 3.6)

Why the Need for Virtual Environments?

There were some interesting decisions made about how and where packages are stored:

• system packages are stored in a child directory of the path stored in sys.prefix
• 3rd party packages are typically placed in one of the directories pointed to by site.getsitepackages

>>> import sys
>>> sys.prefix
'/home/jerry/anaconda3'
>>> import site
>>> site.getsitepackages()
['/home/jerry/anaconda3/lib/python3.7/site-packages']
>>> 

Libraries are stored according to just their name, there is no differentiation between versions.
This is where virtual environments and the virtualenv/venv tools come into play…

Installing virtualenv

If you are using Python 3.3 or newer, the venv module is the preferred way to create and manage virtual environments. venv is included in the Python standard library and requires no additional installation.

virtualenv is used for versions < 3.3. Using virtualenv avoid installing Python packages globally which could break system tools or other projects.

You can install virtualenv using pip:

python3 -m pip install --user virtualenv

Creating a virtual environment

Before you can start installing or using packages in your virtual environment you’ll need to activate it.
Activating a virtual environment will put the virtual environment-specific python and pip executables into your shell’s PATH.

Example: Switching between Python 2 and Python 3 environments

• Start by making a new directory to work with

cd
mkdir python-virtual-environments 
cd python-virtual-environments

• Create a Python environment
• Python 2
Create a new virtual environment inside the directory:

virtualenv py2

Create a Python 2 environment named py2, install Python 2.7 via conda:

conda create --name py2 python=2.7

• Python 3
Create a new virtual environment inside the directory:

python3 -m venv py3

Note, from Python 3.3 to 3.4, the recommended way to create a virtual environment was to use the pyvenv . Create a new environment named py3, install Python 3.5:

conda create --name py3 python=3.5

The creates directory containing a directory structure similar to this:

py3
├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── easy_install
│   ├── easy_install-3.6
│   ├── pip
│   ├── pip3
│   ├── pip3.6
│   ├── python -> python3
│   └── python3 -> /usr/bin/python3
├── include
├── lib
│   └── python3.6
│       └── site-packages
├── lib64 -> lib
├── pyvenv.cfg
└── share
    └── python-wheels

Here’s what each folder contains:
• bin
files that interact with the virtual environment
• include
C headers that compile the Python packages
• lib
a copy of the Python version along with a site-packages folder where each dependency is installed
• Activate the created virtual environment
In order to use this environment’s packages/resources in isolation, you need to “activate” it. To do this, just run the bin/activate script to set up your shell to use the environment’s Python executable and its site-packages by default.
• Python 2

source activate py2

• Python 3

source activate py3

Notice how your prompt is now prefixed with the name of your environment. This is the indicator that env is currently active, which means the python executable will only use this environment’s packages and settings.

(base) jerry@jerry-Latitude-E6410:~/python-virtual-environments$ source py3/bin/activate
(py3) (base) jerry@jerry-Latitude-E6410:~/python-virtual-environments$ 

After activating the environment, we’re now getting a different path for the python executable because, in an active environment, the $PATH environment variable is slightly modified.
• Deactivate the virtual environment
• Python 2

source deactivate

• Python 3

source deactivate

There is a tool virtualenvwrapper created to help to manage many virtual environments. It’s just some wrapper scripts around the main virtualenv tool.

Installing packages

Python has a very active community that contributes an even bigger set of packages that can help you with your development needs. These packages are published to the Python Package Index, also known as PyPI (pronounced Pie Pea Eye). PyPI hosts an extensive collection of packages that include development frameworks, tools, and libraries.

PyPI hosts a very popular library to perform HTTP requests called requests.
You use pip with an install command followed by the name of the package you want to install. pip looks for the package in PyPI, calculates its dependencies, and installs them to insure requests will work.
pip allows you to specify which version of a package to install using version specifiers. For example, to install a specific version of requests:

pip install requests==2.18.4

You can use the list command to see the packages installed in your environment:

$ pip list
Package                            Version 
---------------------------------- --------
absl-py                            0.7.1   

Now that you’re in your virtual environment you can install packages.