Conversation
grouville
force-pushed
the
grouville/reuse-git-advertisement
branch
2 times, most recently
from
September 30, 2026 19:42
a57a6e3 to
1fd898f
Compare
grouville
added a commit
to grouville/dagger
that referenced
this pull request
Sep 30, 2026
Signed-off-by: Guillaume de Rouville <guillaume@dagger.io>
grouville
marked this pull request as ready for review
September 30, 2026 19:45
Module sources are usually written without a URL scheme (for example `github.com/dagger/go`). For each one, `git()` first checks anonymously whether the repository is public, which lists its refs, then lists the same refs again (`git ls-remote`) to confirm it can access the URL it picked. So every remote module was contacted twice per command. When the public check succeeds, keep the refs it received and store them as the repository's ref listing for the rest of the session, so the second listing is skipped. Only anonymous HTTP(S) remotes are reused: remotes with credentials, SSH, a service binding or a user in the URL still list their refs themselves. Access is still checked on every command, because a new session starts empty. On greetings-api (11 remote modules, measured on main plus other open perf PRs), a warm `dagger check -l --all` went from 2.93 s to 2.03 s (median, faster in 10/10 pairs), with no `git ls-remote` left (11 before) and the same 13 public checks. Signed-off-by: Guillaume de Rouville <guillaume@dagger.io>
Signed-off-by: Guillaume de Rouville <guillaume@dagger.io>
Signed-off-by: Guillaume de Rouville <guillaume@dagger.io>
Keep peeled refs immediately after their tag, matching native Git ordering and metadata digests. Extend the real HTTP fixture with v1 and v10 annotated tags. Return the advertisement directly from public probes and remove the two profiling-only hooks. Signed-off-by: Guillaume de Rouville <guillaume@dagger.io>
grouville
force-pushed
the
grouville/reuse-git-advertisement
branch
from
October 1, 2026 19:38
c1d08ea to
ee9eeaf
Compare
vito
approved these changes
Oct 2, 2026
sipsma
approved these changes
Oct 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A schemeless public module source first probes anonymous access, which already retrieves Git's advertised refs. The selected remote then ran
git ls-remoteto retrieve the same refs again. Retain the probe's advertisement in the existing session-owned metadata cache and skip that second request.The probe returns the advertisement directly; nil means the repository requires authentication. No additional visibility-result type or background worker is needed. The two investigation-only
wcprofhooks have been removed.Why the remaining code is needed
Go-git's high-level
ListContextcan invent a symbolic HEAD when the server did not advertise one, and its default result omits peeled tags. Reuse requires the actual advertisement: HEAD, branches, tags, peeled tags and advertised symbolic refs. The small transport/conversion helper retains those values and the existing visibility-error classification.Native Git places a peeled ref immediately after its tag. Stable ordering by the unpeeled name preserves that order and the metadata digest, including prefix names such as
v1andv10. The previous implementation's full-name sort failed this real-Git regression; the corrected ordering passes.Only anonymous HTTP(S) remotes can seed the cache. Credentials, SSH, service bindings and URL user information retain their existing lookup path. Session ownership, the existing credential-scoped key and JSON decoding preserve freshness across commands and give each caller its own mutable copy. Existing or in-flight metadata wins over a new seed.
Performance
Fresh measurements of production code on head
ee9eeafdb1, rebased onto main7916db8d35:Eight alternating A/B and B/A pairs, three fresh sessions per sample, Go 1.26.6 and four Go workers. Every pair was faster; the median paired saving was 129.5 ms. The baseline uses the unchanged current-main visibility probe and a native Git ref listing. Both variants are checked against the exact native ref digest, including annotated prefix tags.
This establishes the avoided request after simplification. The delay is controlled, and the baseline also starts a Git subprocess: it does not establish a fixed saving or a fresh whole-engine command speedup. The earlier stacked-engine command timing table is removed.
Validation
The real HTTP regression verifies that 16 clients in one session make one request, observe exactly the native Git refs and digest, and cannot mutate another caller's copy. It also checks service-bound bypass and a new session observing a repository that becomes private. Scope tests cover username, token, header, SSH socket, URL user information and service bindings. A separate regression verifies that an unadvertised symbolic HEAD is not invented.
The focused public-remote, metadata-cache and priming-scope tests passed three times with the race detector on the rebased, simplified code. Commits are signed off and include a changelog fragment.
CI validation on head
ee9eeafdb1: all 87 Cloud checks are successful. Failed checks were retried with at most two PRs active at a time.