python-dev-tips, Slides

Tutorial first given at ENS Ulm (January 2023) on how to develop a Python package. There is a slight focus on scientific computing, but the general principles apply to any Python project.

With Anaconda (recommended). After installing Anaconda or Miniconda (light version), create a new environment:

# create new environment, press enter to accept
conda create -n project_env python=3.9

# view available environments
conda info --envs

# activate environment
conda activate project_env

# instal package locally
(project_env) pip install -e .

# deactivate environment
(project_env) conda deactivate

For machines really light on memory (e.g. Raspberry Pi) use Virtualenv:

# install library if not already
pip install virtualenv

# create virtual environment (creates folder called project_env)
python3 -m venv project_env

# activate virtual environment
source project_env/bin/activate

# instal package locally
(project_env) pip install -e .

# deactivate virtual environment
(project_env) deactivate

Through pre-commit hooks:

# inside virtual environment
(project_env) pip install pre-commit
(project_env) pip install black

# Install git hooks
(project_env) pre-commit install
# pre-commit installed at .git/hooks/pre-commit

Write tests in the tests folder as function that begin with test_.

To run tests (install pytest first if not already done):

# inside virtual environment
(project_env) pip install pytest

# run tests
(project_env) pytest

To run a specific test:

# inside virtual environment
(project_env) pytest tests/test_fftconvolve.py::test_fft

Will be using Sphinx to generate documentation.

1. Write docstrings when coding. Great if already started!

First time only steps #. (Recommended) create a lightweight virtual environment for building the documentation. Can save a lot of time when building docs remotely, as we'll show with ReadTheDocs

# create new environment, press enter to accept
conda create -n docs_env python=3.9

# activate environment
conda activate docs_env

# install requirements
(docs_env) pip install -e .
(docs_env) cd docs
(docs_env) pip install -r requirements.txt

1. Use sphinx-quickstart (first-time only!). Do separate source and build. Enter project details through terminal.

Otherwise

1. Fill in index and create other files. Edit conf, like adding mock docs.
2. Build docs with e.g. make html.
3. open docs/build/html/index.html in browser.
4. host on ReadTheDocs or GitHub pages.

• take a look at our conf.py which we've modified from the generated one
• ex: changing theme to RTD
• intersphinx mapping to connect to other docs
• mock imports

# create new environment, press enter to accept
conda create -n docs_env python=3.9

# activate environment
conda activate docs_env

# install requirements
(docs_env) pip install -r docs/requirements.txt

1. Will be using Sphinx to generate documentation. This requires

# build docs
(docs_env) python setup.py build_sphinx

1. Use Sphinx to generate documentation.

Uploading to PyPi is done via twine.

In the steps below and after merging to main, replace "X.X.X" with the appropriate version number.

See Semantic Versioning for recommendations on picking version numbers.

# inside virtual environment
(project_env) pip install twine

# edit version in setup
# build package
(project_env) python setup.py sdist bdist_wheel
# -- creates zip in dist folder

# upload to pypi
(project_env) python -m twine upload  dist/pydevtips-X.X.X.tar.gz
# -- X.X.X is the version number in setup.py
# -- enter username and password
# -- check https://pypi.org/project/pydevtips/X.X.X/

# new tag on GitHub
git tag -a X.X.X -m "version X.X.X"
git push origin X.X.X

On GitHub create a new release by:

1. Clicking (the rightmost) "..." dropdown menu.
2. Selecting "Create release".
3. At the bottom pressing "Publish release".

• joblib example in profile
• github page
• point out features in scripts: object-oriented, asserts, tqdm, type hints
• matplotlib, pytest, black in dev install
• example file with hydra
• manifest file to not include file in package
• GitHub actions for releasing to PyPi when changes to version
• documentation (autodoc)
• adding badges to README