Skip to content

virtual-env

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

virtual-env

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
README.md
README.md
 
 

README.md

Overview

That section explains how to install and configure Python virtual environments. It is part of the Python induction for Machine Learning (ML) project, but may be referenced by many other projects.

An alternative way to install Python virtual environments, and software pieces such as Jupyter Lab, is to launch ready-to-use Docker images for Python Jupyter Lab notebooks.

See also

Installation

As the way to manage virtual environments in Python is still not fully stable, we explain here a robust way to use those Python virtual environments on any platform.

Introduction to Pyenv

Pyenv allows to install in parallel on any platform any version of Python, from old deprecated ones to the most recent ones, including the development versions.

Pyenv is packaged on some platforms (e.g., MacOS), but needs to be installed manually on some others (e.g., CentOS). The procedure of installation for pyenv is given below.

Pyenv installs the Python binaries into a folder structure local to the user (typically, ~/.pyenv/), so that they do not interfere with the system-wide Python version.

Introduction to pipenv

pipenv is now an official part of Python and integrates well with virtualenv and pyenv, even though that latter is not (yet?) part of the official Python distribution.

pipenv allows to install in parallel, for any specific project, the full stack of Python dependencies (e.g., NumPy) needed by those projects.

pipenv installs the Python dependencies into virtualenv directories (typically, in ~/.local/share/virtualenvs/), one per specific project or directory, and manages those dependencies through human-readable text files (Pipfile and Pipfile.lock), which can be added to the projects.

Installation of pyenv

The Python binaries are installed in ~/.pyenv, where pipenv can find them.

Dependencies - CentOS 7

  • (As root,) If needed, enable the EPEL repository:
$ sudo yum -y install epel-release
  • (Still as root,) Install a few dependencies:
$ sudo yum -y install git gcc zlib-devel bzip2-devel readline-devel sqlite-devel openssl-devel libffi-devel

Dependencies - Ubuntu / Debian

  • (As root,) Install a few dependencies:
$ sudo aptitude install zlib1g-dev libbz2-dev libsqlite3-dev libreadline-dev libssl-dev libffi-dev

All Linux distributions

  • (As a standard user,) Grab the the latest pyenv source tree from its Github repository and install it in ~/.pyenv:
$ git clone https://github.com/pyenv/pyenv.git $HOME/.pyenv
  • The environment variable PYENV_ROOT needs to be set, and $PYENV_ROOT/bin added to PATH. shims then has to be enabled, as well as relevant auto-completion:
$ cat >> ~/.bashrc << _EOF

## Python pyenv
export PYENV_ROOT="\${HOME}/.pyenv"
export PATH="\${PYENV_ROOT}/bin:\${PATH}"

if command -v pyenv 1>/dev/null 2>&1; then
  eval "\$(pyenv init -)"
fi
_EOF
$ . ~/.bashrc

MacOS

$ brew install pyenv

All: Install a specific version of Python

  • Check which versions of Python 3.7 are available to pyenv to install:
$ pyenv install --list | cut -d' ' -f3,3 | grep -e "^3.7"
3.7.0
3.7-dev
3.7.1
3.7.2
3.7.3
3.7.4
3.7.5
  • Install the version 3.7.4 of Python (latest stable version, as of November 2019):
$ pyenv install 3.7.4
Downloading Python-3.7.4.tar.xz...
-> https://www.python.org/ftp/python/3.7.4/Python-3.7.4.tar.xz
Installing Python-3.7.4...
Installed Python-3.7.4 to ~/.pyenv/versions/3.7.4

Installation of pipenv

Linux

Debian / Ubuntu

$ apt-get install python3-pip python3-all
$ pip3 install --user -U pipenv

CentOS

  • As a standard user:
$ pip install --user -U pipenv

Fedora

$ dnf -y install pipenv

MacOS

$ brew install pipenv

Install specific versions in a given project

  • Go to a project directory and install the required libraries. It will create a Pipfile file, if not already exising.
$ mkdir ~/dev/geo && cd ~/dev/geo
$ git clone https://github.com/opentraveldata/quality-assurance.git opentraveldata-qa
$ cd opentraveldata-qa
$ ./mkLocalDir.sh 
$ pipenv install
Creating a virtualenv for this project…
Pipfile: ~/dev/geo/opentraveldata-qa/Pipfile
Using ~/.pyenv/versions/3.7.4/bin/python3 (3.7.4) to create virtualenv…
⠋ Creating virtual environment...Using base prefix '~/.pyenv/versions/3.7.4'
New python executable in ~/.local/share/virtualenvs/opentraveldata-qa-JhZpfQQq/bin/python3
Also creating executable in ~/.local/share/virtualenvs/opentraveldata-qa-JhZpfQQq/bin/python
Installing setuptools, pip, wheel...done.
Running virtualenv with interpreter ~/.pyenv/versions/3.7.4/bin/python3

✔ Successfully created virtual environment! 
Virtualenv location: ~/.local/share/virtualenvs/opentraveldata-qa-JhZpfQQq
Installing dependencies from Pipfile.lock (a65489)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

Launch Python scripts with pipenv

  • Check that the Python environment is ready
$ pipenv check
Checking PEP 508 requirements…
Passed!
Checking installed package safety…
All good!
  • Just prefix the commands with pipenv run:
$ pipenv run checkers/check-por-cmp-optd-unlc.py

Java

Java is needed for at least a few big data related packages, for instance Hive. The OpenJDK 8 should be good enough for Spark and PySpark.

Fedora

$ sudo dnf -y install java-1.8.0-openjdk-devel java-1.8.0-openjdk-javadoc

Support for graphical frameworks

  • Add a Shell script function for graphical packages (eg, Matplotlib) in ~/.bashrc:
$ cat >> ~/.bashrc << _EOF

# Python - support for Matplotlib in virtual environment
# http://matplotlib.org/faq/osx_framework.html#osxframework-faq
function frameworkpython {
        PYVER=$(python --version | cut -d' ' -f2 | cut -d'.' -f1)
        PYTHON_EXEC=/usr/local/bin/python$PYVER
        if [[ ! -z "${VIRTUAL_ENV}" ]]
        then
                PYTHONHOME=${VIRTUAL_ENV} ${PYTHON_EXEC} "$@"
        else
                ${PYTHON_EXEC} "$@"
        fi
}

_EOF
  • Any Matplotlib-based Python script can then be launched with:
$ frameworkpython myPyscript.py

General - A few useful Python packages

An arbitrary selection of useful packages.

$ pipenv install cython
$ pipenv install networkx
  • Graphviz, graph visualization software:
$ pipenv install graphviz
  • Seaborn, statistical data visualization:
$ pipenv install seaborn
  • imageio, Python library for reading and writing image data:
$ pipenv install imageio
$ pipenv install altair vega_datasets gpdvega
$ pipenv install matplotlib cartopy neobase
  • xlrd, working with Excel Files in Python:
$ pipenv install xlrd
$ pipenv install numpy scikit-learn pandas dask
  • ml-tools, a collection of ML tools for object detection and classification on Digital Globe (DG) imagery:
$ pipenv install ml-tools
  • Tensorflow, an open source ML framework for everyone:
$ pipenv install tensorflow
  • Keras, the Python Deep Learning (DL) library:
$ pipenv install keras scikit-keras
$ pipenv install Theano
  • gym, a toolkit for developing and comparing reinforcement learning (RL) algorithms:
$ pipenv install gym
  • PyTorch, an open source deep learning (DL) platform that provides a seamless path from research prototyping to production deployment:
$ pipenv install pytorch
$ pipenv install onnxmltools
$ pipenv install sklearn-pmml-model
  • surprise, a Python scikit building and analyzing recommender systems:
$ pipenv install surprise
  • parfit, package for parallelizing the fit and flexibly scoring of Scikit-learn ML models, with visualization routines:
$ pipenv install parfit
  • mcfly, a deep learning (DL) tool for time series classification:
$ pipenv install mcfly
$ pipenv install featuretools
$ pipenv install pyspark
$ pipenv install elasticsearch_opentracing
  • Install latest versions (e.g., of pyproj and basemap):
$ pipenv install git+https://github.com/jswhit/pyproj.git#egg=pyproj \
         git+https://github.com/matplotlib/basemap.git#egg=basemap
$ pipenv install jupyterlab
  • Various Jupyter modules:
$ pipenv install jupyter-git jupyter-pip jupyter-beautifier jupyter-full-width jupyter-notebook-gist
$ pipenv install jupyter_dashboards jupyter_dashboards_bundlers jupyter-spark
$ pipenv install jupyter_utils jupyter-tools
  • Jupyter extensions:
$ pipenv install jupyter_contrib_nbextensions
$ jupyter contrib nbextension install --user
$ jupyter labextension install jupyterlab-drawio
$ pipenv install bash_kernel ipykernel r2_kernel

Typical session

pyenv

  • Check the installed versions of Python:
$ pyenv versions
* system (set by ~/.pyenv/version)
  3.7.4

pipenv

  • Prefix any Python command/script with pipenv run:
$ mkdir -p /tmp/$USER && cd /tmp/$USER
$ pipenv install --python 3.7
Creating a virtualenv for this project…
Pipfile: /tmp/$USER/Pipfile
Using ~/.pyenv/versions/3.7.1/bin/python3.7 (3.7.4) to create virtualenv…
⠹Running virtualenv with interpreter ~/.pyenv/versions/3.7.4/bin/python3.7
Using base prefix '~/.pyenv/versions/3.7.4'
New python executable in ~/.local/share/virtualenvs/$USER-bV5bYO4h/bin/python3.7
Also creating executable in ~/.local/share/virtualenvs/$USER-bV5bYO4h/bin/python
Installing setuptools, pip, wheel...
done.

Virtualenv location: ~/.local/share/virtualenvs/$USER-bV5bYO4h
Creating a Pipfile for this project…
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (a65489)!
Installing dependencies from Pipfile.lock (a65489)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
  • Use Python without activating the virtual environment:
$ cd /tmp/$USER
$ pipenv run python --version
Python 3.7.4
  • Use Python in an activated virtual environment:
$ cd /tmp/$USER
$ pipenv shell
($USER) $ python --version
Python 3.7.4
($USER) $ exit

PySpark

  • PySpark, run from the Python virtual environment, needs a standalone Java Virtual Machine (JVM) to run Spark. So, be sure to install Java 8, as explained in the above installation section.

  • Launch a Spark Shell:

$ pipenv run spark-shell
[...]
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[*], app id = local-12345).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.0
      /_/
         
Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_181)
Type in expressions to have them evaluated.
Type :help for more information.

scala> spark.version
res0: String = 2.4.0

scala> :quit
$