Skip to content

Deduplicate Git fetches for subdirectory packages from the same repository and commit #14725

Description

@AdrianDuan

Describe the user story

When consuming multiple packages from the same Git monorepo, pinned to the same commit but using different &path: selectors, I would like pnpm to fetch the repository once per installation and reuse that source for the individual packages.

For example (illustrative placeholders, not a runnable fixture):

{
  "dependencies": {
    "@example/sdk": "git+ssh://git@github.com/example/monorepo.git#<full-commit-sha>&path:/packages/sdk",
    "@example/contract": "git+ssh://git@github.com/example/monorepo.git#<full-commit-sha>&path:/packages/contract"
  }
}

With an empty package cache, separate repository fetches are costly for large monorepos, particularly private repositories accessed over SSH. Package-level warm-cache reuse helps subsequent installs, but does not address fetching different subdirectory packages from the same source during a cold install.

Versions inspected: pnpm 11.9.0 (locally installed implementation) and pnpm 12.4.0 (source review only; I have not run a pnpm 12 reproduction).

In the v12.4.0 Git fetcher, run_sync creates a temporary directory and calls checkout_into before preparing the selected subdirectory. checkout_into performs a shallow fetch or clone. From this code, source acquisition appears to remain per Git package rather than shared across subdirectory packages. Please correct me if an upstream layer already coalesces these fetches.

Related: #7483 added Git subdirectory support. This request concerns sharing source acquisition across different subdirectories, not adding that syntax.

Describe the solution you'd like

Add install-scoped deduplication of Git source acquisition for resolutions with the same repository URL and resolved commit, irrespective of the package subdirectory.

A bounded first implementation could:

  • Share an in-flight fetch and immutable source snapshot for an identical repository URL + full commit SHA within one install.
  • Materialize separate working directories for each package's preparation/build, so scripts cannot mutate another package's source or race with it.
  • Keep package identities, subdirectory validation, package contents, and package-store entries separate.
  • Preserve authentication and transport boundaries; URL normalization across different credentials or transports is not required.
  • Clean up temporary source data after success or failure. Persistent cross-install Git caching can be a separate enhancement.

Suggested acceptance checks using an isolated store and a Git command wrapper:

  1. Two packages at different paths in the same repository and commit trigger one source fetch/clone during a cold install.
  2. Concurrent requests share that operation rather than starting duplicate downloads.
  3. Both installed packages have the correct manifests and files; independent prepare scripts cannot contaminate each other.
  4. Different repositories or commits remain correctly isolated.
  5. Failed source acquisition is propagated to waiting consumers and does not leave a reusable partial snapshot.
  6. Warm package-cache installs retain their existing behavior.

Describe the drawbacks of your solution

This introduces concurrency coordination and temporary-source ownership/cleanup. Per-package isolated working directories may still consume additional disk space or copying time. Preparation scripts can depend on repository context, so sharing a mutable checkout would be unsafe. Authentication context must not be accidentally broadened by cache-key normalization.

Describe alternatives you've considered

  • Rely on the existing package store: useful after each package has been cached, but it does not share the first acquisition across different paths.
  • Shallow fetch: reduces history transfer, but each package can still fetch the same commit independently.
  • Publish registry packages or tarballs: avoids repository cloning, but changes the distribution workflow and is not always available.
  • Merge packages or duplicate generated contract sources into the SDK: can reduce the number of Git dependencies, but changes package boundaries to work around installation behavior.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions