Cache restoration is orders of magnitude slower on Windows compared to Linux.

We are using this action to cache Conan packages.

Our current cached is ~300 MB on both Linux (ubuntu-20.04) and Windows (windows-2019).
Sadly, where the cache step routinely takes ~10 seconds on Linux, it oftentimes takes ~5 minutes on Windows.

This makes iterations frustrating, as the rest of the workflow takes about 2 minutes to complete, we get a global x3 time penalty because of this.

As far as I can tell, the archive retrieval time is comparable, it really is the cache un-archiving which seems to take very long on Windows.

Our latest run below for illustration purposes.

Linux:

Windows:

really is the cache un-archiving which seems to take very long on Windows.

That's because tar is extremely slow on Windows. Some related links:

Thank you for your reponse!

(If the issue is with the specific un-archiving utility, maybe there is an alternative that the action could use on Windows to get better performances?)

I'm experiencing the same issue. Here is one of our latest runs after implementing dependency caching.

https://github.com/getsentry/sentry-dotnet/runs/6327391846?check_suite_focus=true

Time to restore dependencies from cache:

Ubuntu	macOS	Windows
18s	29s	4m 5s

With timings visible, it's clearly the tar operation that is the culprit:

@bishal-pdMSFT

Looks like this has been going on for awhile... See also #442 and #529. Hopefully @bishal-pdMSFT can make some improvements here? Maybe just provide an optional parameter to the action that would tell it to use .zip (or another format) instead of .tgz on Windows? 7-Zip is pre-installed on the virtual environments.

Run steps that must be run only once on ubuntu-latest

@vsvipul

@vsvipul To restore the cache, the ubuntu server takes 16 seconds, and the windows server takes 2 minutes, and 29 seconds.
https://github.com/space-wizards/space-station-14/runs/7017334343?check_suite_focus=true

@Adnn

Hi @Adnn @wrexbe @mattjohnsonpint, we do understand. Actually zstd is disabled on Windows due issues with bsd tar, hence the slowness of restore. While we look into it, you could use the workaround suggested here to improve your hosted windows runner performance. This would enable zstd while using GNU tar.

TLDR; Add the following step to your workflow before the cache step. That's it.

- if: ${{ runner.os == 'Windows' }}
      name: Use GNU tar
      shell: cmd
      run: |
        echo "Adding GNU tar to PATH"
        echo C:\Program Files\Git\usr\bin>>"%GITHUB_PATH%"

Hope this helps.

@pdotl Thank you for your response. It is encouraging to read this issue will be looked into!

On the other hand, I tried to add the Use GNU tar step you provided, just before the cache action in our pipeline. The step ran and correctly output "Adding GNU tar to PATH", yet I did not observe any noticeable speed-up (but as I am not used to do testing at the workflow level, I may be proven wrong).

@Adnn

@pdotl - Thanks. But like @Adnn, I have to report that changing to gnu tar gave no significant performance gain.

https://github.com/getsentry/sentry-dotnet/runs/7979994529?check_suite_focus=true#step:3:58

Tue, 23 Aug 2022 18:07:43 GMT Cache Size: ~1133 MB (1187773080 B)
Tue, 23 Aug 2022 18:07:43 GMT "C:\Program Files\Git\usr\bin\tar.exe" --use-compress-program "zstd -d" -xf D:/a/_temp/6caa1415-0002-48ab-a4c2-b1ac4df21961/cache.tzst -P -C D:/a/sentry-dotnet/sentry-dotnet --force-local
Tue, 23 Aug 2022 18:13:01 GMT Cache restored successfully

As you can see from the timestamps, it took over 5 minutes to decompress. The same build on Linux and macOS took less than 30 seconds.

I realize that it would require some significant changes to actions/toolkit, but I think to really get comparable perf we're going to need ability to use a format other than tar.gz. Perhaps .7z since 7-Zip is pre-installed on the runners?

Use gnu tar on windows CI

VisualStudioHelpers.ps1 stop enabling 8.3 filename creation

@samypr100

@samypr100 Your Github Action is awesome! And the examples are super useful!
Thanks for creating it and also for the acknowledgements in the README 🙂

I think that, if something similar would be implemented in this Action and the documentation was updated, this issue could finally be closed (after more than 2 years)...

VisualStudioHelpers.ps1 stop enabling 8.3 filename creation (#9862)

@samypr100

Awesome job @samypr100 - I'll have to try it out! - I was experimenting here with NTFS and tried ReFS - https://github.com/malkia/opentelemetry-cpp/blob/main/.github/workflows/otel_sdk.yml#L28 - though the VHDX gets to be much bigger (in size, granted mostly zeroes). Wonder what else I can do there.

@malkia

I also experimented with ReFS on my local Windows machine and the performance (even when declared as a DevDrive with AntiVirus turned off) is significantly worse (slower and, as @malkia says, requires more space) than NTFS for my use-case (Bazel cache).

Thats unfortunate to hear, the action does give you the flexibility to change the format to NTFS if desired.

This would completely derail the conversation, but I'm evaluating ReFS / DevDrive at work, and we used a lot of reparse points for our custom source file system. It turns out in ReFS 3.10 (the one shipped today I think in Win11) this reparse point is somewhere between 8-16kb (can't tell, I've empirically derived this number), but then I switched for week to Insiders edition, and there it was ReFS 3.14 - and this was down to less than thousand bytes (per our special link).

Now the bazel cache have none of that (the disk cache), it's just a CAS, but ReFS is improving at fronts, so I guess it's worth re-evaluating it later. For our use case, we are trying to limit the amount of AV scanning and other minifilters (though we had to explicitly add our minifilter in the loop for things to work). It supports block-level (not file-level!) deduplication - so if you have situation where a build copies over and over .dll/.exe files this would be great to have!

ci: speed up windows build (#3474)

feat(cache): opt-in env var to prefer BSD tar on Windows

Root-caused and a fix is up at actions/toolkit#2379 — adds an opt-in env var ACTIONS_CACHE_PREFER_BSD_TAR_ON_WINDOWS that switches the archiver from Git-for-Windows' MSYS GNU tar to C:\Windows\System32\tar.exe (libarchive bsdtar).

The bottleneck isn't the cache transport and isn't (primarily) Defender — it's the tar itself. MSYS tar pays a POSIX-shim translation cost on every file and forks an external zstd.exe process per stream, which adds up fast on many-small-files payloads. bsdtar makes native Win32 syscalls and has zstd built in.

Fresh numbers on windows-2025 hosted runner, 568 MB / ~40k-file cache:

archiver	create (s)	extract (s)	archive (MB)
MSYS tar + zstd (current default)	16–20	~80	567
bsdtar + zstd (proposed opt-in)	14–16	16–19	568

~4× faster extract, no size penalty. Also tested 7z (LZMA2 -mx3) for completeness — extract ~41 s, archive 28% larger; strictly worse than bsdtar.

Windows Defender exclusion on the target dir only shaves ~12 s off the MSYS path and near-zero off bsdtar — so AV isn't the dominant factor here.

PR is opt-in (env var) to keep risk low; once it soaks in a few consumers we could revisit flipping the default.

Improve cache restoration on Windows

ci: put Windows Go caches on the fast work disk via a setup-go-cache …

ci: restore-only toolchain cache on the acceptance shards (cloudposse…

Navigation Menu

Repository navigation

Cache restoration is orders of magnitude slower on Windows compared to Linux. #752

23 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Cache restoration is orders of magnitude slower on Windows compared to Linux. #752

Description

Activity

rikhuijzer commented on Mar 6, 2022

Adnn commented on Mar 7, 2022

mattjohnsonpint commented on May 6, 2022

mattjohnsonpint commented on May 9, 2022

wrexbe commented on Jun 23, 2022

lvpx commented on Aug 22, 2022

Adnn commented on Aug 23, 2022

mattjohnsonpint commented on Aug 23, 2022

23 remaining items

paco-sevilla commented on May 16, 2024

malkia commented on May 16, 2024

paco-sevilla commented on May 17, 2024

samypr100 commented on May 17, 2024

malkia commented on May 17, 2024

zeitlinger commented on Apr 17, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions