PIP cache should cache the installed packages as well

Description:
Currently, setup-python caches only the ~/.cache/pip directory to avoid redownloads. However, it doesn't cache the installed packages. As some package have lengthy installation steps, this leads to delays in builds.

You can see the current behaviour for example in https://github.com/crabhi/setup-python-cache-test/actions/runs/1789016634 (or in attached build.txt) - the pip install output shows "Collecting" and "Installing" instead of "Requirement already satisfied" for all packages.

Justification:
For example installing the ansible package takes well over a minute even if it's already downloaded.

Are you willing to submit a PR?
Yes, I can try.

@crabhi

Hello @crabhi, thanks for your request!
We will look at it.

would be also nice to follow https://github.com/actions/cache#outputs and provide an output cache-hit so we can if: steps.[id]. cache-hit != 'true' to avoid calling pip altogether.

which is done at https://github.com/actions/cache/blob/2d8d0d1c9b41812b6fd3d4ae064360e7d8762c7b/src/utils/actionUtils.ts#L25-L27 and https://github.com/actions/cache/blob/main/src/restore.ts#L55

This pattern affects more languages (actions/setup-node works the same - only caches downloads, not installs) - would love to see a general consensus towards caching installs, not tarballs (perhaps behind a flag/attribute for future compat cache: pip-install).

for pip I think it makes even more sense as there are no postinstall actions... with setup-node I'm also caching the node_modules instead of the packages, but it "broke" some flows where there was a postinstall script to configure other things (like pre-build some typescript scripts). The solution is simple, just run that script manually (or in my case, cache the built scripts)... but not "one size fits all".

For pip AFAIR there are no postinstall scripts, then this would not be an issue.

For pip AFAIR there are no postinstall scripts, then this would not be an issue.

I'm experimenting with this at the moment and caching site-packages (read: pip output) isn't straightforward either; for instance binary wrappers (black, ..) won't work (python -m black works fine tho). Might be one of thos YMMV cases that makes it hard to standardize for everyone.

would be also nice to follow https://github.com/actions/cache#outputs and provide an output cache-hit so we can if: steps.[id]. cache-hit != 'true' to avoid calling pip altogether.

Hey, this feature was merged today and should be a part of the near-future release

but the cache-hit is just for the packages, not the installation, right? IOW: do I still need to call pip install?

but the cache-hit is just for the packages, not the installation, right? IOW: do I still need to call pip install?

Oh you're talking pip. Well yeah, then you'll have to wait for this action to support caching venv's out of the box. It's a case for pipenv and poetry though. The best this for now is to manually cache

I have a case where building packages for pypy (grpcio, grpcio-tools) takes about 6 minutes-- it's way too slow to introduce a matrix.

If anyone has a manual example using actions/cache, please share it.

I was creating a python venv and then caching that directory, however I hit an issue where that was broken once restored (behaviour was inconsistent).

I currently have a job that takes ~6 min to complete, 4 min of which is installation of pip packages. An effective caching of installed packages would be a great boost.

I was creating a python venv and then caching that directory, however I hit an issue where that was broken once restored (behaviour was inconsistent).

I currently have a job that takes ~6 min to complete, 4 min of which is installation of pip packages. An effective caching of installed packages would be a great boost.

Could you share the workflow so the people can take a look at it? I think it's possible to hack around while this feature is not here

CI: always run pip install, even when cache found

I wonder if it's actually worth it.
Here I cached the content of ${{ env.pythonLocation }}/lib/site-packages and ${{ env.pythonLocation }}/Scripts using actions/cache:

No caching:

Caching, no cache hit (+1m):

Caching, cache hit (+18s):

@Avasam

@Avasam possibly at least less strain on pypi. Also we should test small and big amounts of dependencies

Just wanted to add an anecdote of my own experience. TorchGeo has a long list of dependencies:

Install times without caching vary quite a bit by OS and Python version:

Python	Linux	macOS	Windows
3.10	2m 30s	2m 23s	5m 4s
3.9	2m 50s	4m 50s	5m 49s
3.8	2m 29s	2m 12s	3m 19s

We first tried using the cache feature of setup-python:

    - name: Set up python
      uses: actions/setup-python@v4.5.0
      with:
        python-version: ${{ matrix.python-version }}
        cache: 'pip'
        cache-dependency-path: |
          requirements/required.txt
          requirements/datasets.txt
          requirements/tests.txt
    - name: Install pip dependencies
      run: pip install -r requirements/required.txt -r requirements/datasets.txt -r requirements/tests.txt

Not only do install times not significantly improve, in many cases it's actually worse!

Python	Linux	macOS	Windows
3.10	2m 42s	1m 53s	5m 50s
3.9	2m 50s	2m 11s	5m 46s
3.8	2m 39s	3m 21s	2m 35s

Finally, we tried the setup proposed in this blog that manually caches the entire Python installation:

    - name: Set up python
      uses: actions/setup-python@v4.5.0
      with:
        python-version: ${{ matrix.python-version }}
    - name: Cache dependencies
      uses: actions/cache@v3.2.4
      id: cache
      with:
        path: ${{ env.pythonLocation }}
        key: ${{ env.pythonLocation }}-${{ hashFiles('requirements/required.txt') }}-${{ hashFiles('requirements/datasets.txt') }}-${{ hashFiles('requirements/tests.txt') }}
    - name: Install pip dependencies
      if: steps.cache.outputs.cache-hit != 'true'
      run: pip install -r requirements/required.txt -r requirements/datasets.txt -r requirements/tests.txt

This resulted in significantly faster installation times, which could likely be further improved by only caching the site-packages directory:

Python	Linux	macOS	Windows
3.10	38s	39s	4m 11s
3.9	53s	45s	4m 20s
3.8	1m 2s	1m 15s	1m 21s

Apparently slower Windows caching is a known issue: actions/cache#752.

So yes, if setup-python also cached installed packages, that would be awesome!

which could likely be further improved by only caching the site-packages directory

In hindsight, this is a bad idea, many tools like black or flake8 also install files into bin so we'll at least need to cache bin too.

In hindsight, this is a bad idea, many tools like black or flake8 also install files into bin so we'll at least need to cache bin too.

I addressed this point a while ago (above) - recap here:

I'm experimenting with this at the moment and caching site-packages (read: pip output) isn't straightforward either; for instance binary wrappers (black, ..) won't work (python -m black works fine tho). Might be one of thos YMMV cases that makes it hard to standardize for everyone.

So, instead of invoking black, do python -m black.

That's a decent workaround, but I don't think it's realistic to expect all users to change how they invoke other steps later in their workflow. I think we would have to cache bin too. Possibly everything. Bonus of caching everything is that we have to install Python from a cache anyway.

That's a decent workaround, but I don't think it's realistic to expect all users to change how they invoke other steps later in their workflow. I think we would have to cache bin too. Possibly everything. Bonus of caching everything is that we have to install Python from a cache anyway.

Most definitely not a catch-all! To be honest I'm not confident there's a straightforward solution..

@adamjstewart

The workaround from @adamjstewart seems to work wonders indeed ! But I think a standard implementation from this repository would be a great addition. Any updates on it from the dev team ?

Just wanted to add an anecdote of my own experience. TorchGeo has a long list of dependencies:

requirements/required.txt

requirements/datasets.txt

requirements/text.txt

Install times without caching vary quite a bit by OS and Python version:

Python Linux macOS Windows
3.10 2m 30s 2m 23s 5m 4s
3.9 2m 50s 4m 50s 5m 49s
3.8 2m 29s 2m 12s 3m 19s
We first tried using the cache feature of setup-python:
    - name: Set up python
      uses: actions/setup-python@v4.5.0
      with:
        python-version: ${{ matrix.python-version }}
        cache: 'pip'
        cache-dependency-path: |
          requirements/required.txt
          requirements/datasets.txt
          requirements/tests.txt
    - name: Install pip dependencies
      run: pip install -r requirements/required.txt -r requirements/datasets.txt -r requirements/tests.txt
Not only do install times not significantly improve, in many cases it's actually worse!

Python Linux macOS Windows
3.10 2m 42s 1m 53s 5m 50s
3.9 2m 50s 2m 11s 5m 46s
3.8 2m 39s 3m 21s 2m 35s
Finally, we tried the setup proposed in this blog that manually caches the entire Python installation:
    - name: Set up python
      uses: actions/setup-python@v4.5.0
      with:
        python-version: ${{ matrix.python-version }}
    - name: Cache dependencies
      uses: actions/cache@v3.2.4
      id: cache
      with:
        path: ${{ env.pythonLocation }}
        key: ${{ env.pythonLocation }}-${{ hashFiles('requirements/required.txt') }}-${{ hashFiles('requirements/datasets.txt') }}-${{ hashFiles('requirements/tests.txt') }}
    - name: Install pip dependencies
      if: steps.cache.outputs.cache-hit != 'true'
      run: pip install -r requirements/required.txt -r requirements/datasets.txt -r requirements/tests.txt
This resulted in significantly faster installation times, which could likely be further improved by only caching the site-packages directory:

Python Linux macOS Windows
3.10 38s 39s 4m 11s
3.9 53s 45s 4m 20s
3.8 1m 2s 1m 15s 1m 21s
Apparently slower Windows caching is a known issue: actions/cache#752.

So yes, if setup-python also cached installed packages, that would be awesome!

This right here has been a life saver for me - I toiled over this caching for so long, but this got me there!! Thank you so so so much!!

@adamjstewart

In hindsight, this is a bad idea, many tools like black or flake8 also install files into bin so we'll at least need to cache bin too.

@adamjstewart, thanks for your awesome post, it really reduced my build times.

It's definitely possible to achieve your idea of caching only the necessary items instead of the entire python build by caching the site-packages and bin (in order to get executable packages like black, ruff, etc..). In my case, my two dependency files are classic pip-tools dev-requirements.txt and requirements.txt.

  - name: Cache dependency libraries
    uses: actions/cache@v3.2.4
    id: cache-libraries
    with:
      path: ${{ env.pythonLocation }}/lib/python3.11/site-packages
      key: ${{ env.pythonLocation }}-${{ hashFiles('dev-requirements.txt') }}-${{ hashFiles('requirements.txt') }}

  - name: Cache dependency binaries
    uses: actions/cache@v3.2.4
    id: cache-binaries
    with:
      path: ${{ env.pythonLocation }}/bin
      key: ${{ env.pythonLocation }}-${{ hashFiles('dev-requirements.txt') }}-${{ hashFiles('requirements.txt') }}

Finally I run the pip install step conditionally based on successful cache hits with the following:

  - name: Install pip-tools and dependencies
    if: steps.cache-libraries.outputs.cache-hit != 'true' || steps.cache-binaries.outputs.cache-hit != 'true'
    run: |
      python -m pip install --upgrade pip
      pip install pip-tools
      pip-sync dev-requirements.txt requirements.txt

Mileage can vary for small dependency lists based on the cache restore network speed.

@adamjstewart

@adamjstewart Thanks for the cache example, it really helps. :-)

I have one more suggestion that I decided to use, because I want to avoid downloading all pip dependencies everytime any of them changes - to introduce a second cache just for pip cache.

            -   name: "Python: Setup Python"
                uses: actions/setup-python@v4.7.0
                with:
                    python-version: 3.10.13

            -   name: "Cache: Cache Python"
                id: python-cache
                uses: actions/cache@v3.3.1
                with:
                    path: ${{env.pythonLocation}}
                    key: ${{env.pythonLocation}}-${{hashFiles('requirements-dev.txt')}}-${{hashFiles('requirements.txt')}}

            -   name: "Shell: Get pip cache dir"
                id: pip-cache-dir
                if: steps.python-cache.outputs.cache-hit != 'true'
                run: |
                    python -m pip install -U pip
                    pip install -U wheel
                    echo "pip-cache-dir=$(pip cache dir)" >> ${GITHUB_OUTPUT}

            -   name: "Cache: Cache pip"
                if: steps.python-cache.outputs.cache-hit != 'true'
                uses: actions/cache@v3.3.1
                with:
                    path: ${{steps.pip-cache-dir.outputs.pip-cache-dir}}
                    key: 3.10-${{hashFiles('requirements-dev.txt')}}-${{hashFiles('requirements.txt')}}
                    restore-keys: |
                        3.10-${{hashFiles('requirements-dev.txt')}}-
                        3.10-

            -   name: "Shell: Install pip dependencies"
                if: steps.python-cache.outputs.cache-hit != 'true'
                run: pip install -r requirements-dev.txt

Unfortunately, the built-in cache functionality only supports dependencies downloaded by package manger, not the whole Python as expected.

I made an example to cache both of download libraries and binaries (python + package manager like pip) via actions/cache

    - name: Cache dependencies
      uses: actions/cache@v3.2.4
      id: cache
      with:
        path: ${{ runner.tool_cache }}/Python/${{ inputs.python-version }} # e.g /opt/hostedtoolcache/Python/3.11.6
        key: ${{ runner.tool_cache }}/Python/${{ inputs.python-version }}/${{ runner.arch }}-${{ hashFiles('requirements.txt') }}

    - name: Set up Python env
      uses: actions/setup-python@v4
      with:
        token: ${{ secrets.xyz }} # if needed
        python-version: ${{ inputs.python-version }}

    - name: Install pip dependencies
      shell: bash
      run: |
        pip install -r requirements.txt

The key is on the cache step standing before the actions/setup-python. It will restored the cache, includes: binaries and libraries then parse into path. actions/setup-python sees x64.complete 0-byte file existed under path so will not download Python binary again.

The console output with cache should be

Cache restored successfully
Cache restored from key: /opt/hostedtoolcache/Python/3.11.6/x64-95421aa4e4a3b4024058ea1f527f5a724a65d3408f2a8c64daad58bfa48fa33d
...
Run actions/setup-python@v4
  Successfully set up CPython (3.11.6)
...
Run pip install -r requirements.txt
Requirement already satisfied: requests==2.25.1 in /opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages (from -r requirements.txt (line 2)) (2.25.1)
Requirement already satisfied: pyyaml==6.0.1 in /opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages (from -r requirements.txt (line 3)) (6.0.1)

Note: the above ^ testing on self-host runner.

Finally, we tried the setup proposed in this blog that manually caches the entire Python installation:

This works pretty well after the cache is populated, however, populating the cache for me seems to take 13m with the deps we have, not just on initial creation, but also when the cache is busted

This is running

/usr/bin/tar --posix -cf cache.tgz --exclude cache.tgz -P -C /home/runner/work/xxx/yyy --files-from manifest.txt -z

Has anyone had any luck with a similar approach that will handle partial restores more cleanly (without using pipenv, which is what I've typically done for caching in the past)

Navigation Menu

Repository navigation

PIP cache should cache the installed packages as well #330

9 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

PIP cache should cache the installed packages as well #330

Description

Activity

nikita-bykov commented on Feb 4, 2022

barbieri commented on Mar 24, 2022

barbieri commented on Mar 24, 2022

jbergstroem commented on Mar 29, 2022

barbieri commented on Mar 29, 2022

jbergstroem commented on Mar 29, 2022

dhvcc commented on Apr 5, 2022

barbieri commented on Apr 5, 2022

dhvcc commented on Apr 5, 2022

belm0 commented on Apr 16, 2022

rashidnhm commented on May 9, 2022

dhvcc commented on May 10, 2022

9 remaining items

Avasam commented on Nov 24, 2022

dhvcc commented on Dec 2, 2022

adamjstewart commented on Feb 5, 2023

adamjstewart commented on Feb 7, 2023

jbergstroem commented on Feb 7, 2023

adamjstewart commented on Feb 7, 2023

jbergstroem commented on Feb 7, 2023

Seluj78 commented on Feb 22, 2023

CoreyGaunt commented on Apr 7, 2023

nicklausbrown commented on Aug 28, 2023

harpener commented on Sep 7, 2023

itsme2980 commented on Oct 11, 2023

wyardley commented on Oct 27, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions