[Bug][gitextractor] Incremental collection permanently drops in-range commits behind merge commits #9189

Search before asking

• I had searched in the issues and found no similar issues.

Related, but closed as stale without a root cause: #8434 (same skip commit ... because it has no parent commit log and the same Lead Time impact), #8818, #8857. This issue adds the root cause and a small self-contained reproduction. It also explains why "increase the time range" (suggested in #8434) does not help: the lost commits are newer than the incremental start time, not older than timeAfter.

What happened

In short: after the first run, gitextractor collects commits incrementally. In some common merge situations, it silently and permanently loses commits that are well inside the collection window. Their child commits are stored, and commit_parents even points to them, but the commits themselves never appear in commits or repo_commits, and later runs never pick them up.

Why it matters: when a lost commit is a deployment head, refdiff compares the next deployment against an incomplete commit graph. Very old PRs are then attributed to a recent deployment, and DORA Lead Time for Changes jumps (we see samples of about 10,000 hours). #9187 describes how those wrong diffs then stay cached.

What we see on our instance (self-managed GitLab, daily blueprints, merge-commit MR workflow):

• In one repository, 7 of 91 default-branch commits over six weeks were never collected. They are scattered, not in one time range.
• At least 35 production deployment head commits have exactly this signature: missing from commits, but referenced as parent_commit_sha in commit_parents.
• The pipeline log shows skip commit <sha> because it has no parent commit for them. Most other skipped SHAs really are older than timeAfter (the intended case), so the log line alone does not reveal the problem.

How it happens, step by step

Take this history on the default branch. The previous incremental run started at 12:00, so this run fetches with since = 12:00:

            12:00 = since (previous run start)
                 |
  R ------------ | -- Q ------ P ------ M      main
  (01-01)        |   13:00    14:00    16:00 (merge commit)
   \             |                     /
    F ---------- | --------------------+       feature (head F at 10:00,
   10:00         |                              collected by the previous run)

Q and P landed on main after the previous run. Then a merge request merged feature (whose head F is older than since) with the merge commit M. So this run should store Q, P and M.

Incremental clone does this (GitcliCloner.shallowClone() + deepen() in plugins/gitextractor/parser/clone_gitcli.go):

git clone --depth=1 --bare <url>
git fetch --depth=1 origin
git fetch --shallow-since=<since>
git repack -d
git fetch --deepen=1

1. --shallow-since stops at M. Git decides "shallow or not" per commit: because one parent of M (F) is older than since, git cuts M off from all its parents. P and Q are newer than since, but they are only reachable through M, so they are not fetched.

   local repo after --shallow-since:   M (shallow: no parents)      F (branch head)
   

2. --deepen=1 adds one generation, so P arrives. Q, the parent of P, is still missing.

   local repo after --deepen=1:        P (parent Q missing) -- M      F
   

3. The collector drops P entirely. Libgit2RepoCollector.CollectCommits() (the same code is in GogitRepoCollector) returns early for any commit whose first parent is not in the local repo:

   // Skip calculating commit statistics when there are parent commits, but the first one cannot be fetched from the ODB.
   ...
   if parent == nil {
       r.logger.Info("skip commit %s because it has no parent commit", commit.Id().String())
       return nil
   }

   This return nil runs before store.Commits, store.RepoCommits and storeParentCommits, so P is not stored at all. The comment (added in cherry-pick #7719 fix: addition of some commits are ridiculous huge #7720) says only the statistics should be skipped. Q was never fetched.

4. The gap is permanent. The next run uses since = the start of this run. P and Q are now older than that, so no incremental run fetches them again. Only a full sync recovers them.

When it triggers: between two runs, the default branch first gets a new commit or merge (P), and then a merge request whose source-branch head is older than the previous run is merged with a merge commit (M). This is common in merge-commit MR/PR workflows. Linear histories (fast-forward or squash only) are not affected.

What do you expect to happen

Incremental collection stores every commit that is reachable from the collected branches and newer than the previous run, together with its repo_commits and commit_parents rows. In the example, this run stores Q, P and M.

How to reproduce

The script below builds the history above in a temporary git repository (git only, no network). It runs the same git commands as GitcliCloner.shallowClone() + deepen() for two consecutive incremental runs. Then it applies the CollectCommits() rule ("skip if the first parent is not in the local repo") to show what would be stored.

#!/usr/bin/env bash
# Synthetic, self-contained reproduction of the gitextractor incremental-clone gap.
# Replays GitcliCloner.shallowClone()+deepen() (plugins/gitextractor/parser/clone_gitcli.go)
# and applies the Libgit2RepoCollector.CollectCommits() "no parent commit" rule, for two
# consecutive incremental runs.
set -euo pipefail
work="$(mktemp -d)"
trap 'rm -rf "${work}"' EXIT
cd "${work}"

commit() { # commit <date> <message> [parents...]
  local date="$1" msg="$2"; shift 2
  local args=(); for p in "$@"; do args+=(-p "$p"); done
  GIT_AUTHOR_DATE="${date}" GIT_COMMITTER_DATE="${date}" \
    git -C origin.git commit-tree "${tree}" "${args[@]}" -m "${msg}"
}

# incremental_run <since> <label>: one gitextractor incremental run into a fresh dir.
incremental_run() {
  local since="$1" label="$2" dir="${work}/$2.git"
  git clone -q --depth=1 --bare "file://${work}/origin.git" "${dir}"
  printf '\tfetch = +refs/heads/*:refs/remotes/origin/*\n' >>"${dir}/config"
  git -C "${dir}" fetch -q --depth=1 origin
  git -C "${dir}" fetch -q --shallow-since="${since}"
  git -C "${dir}" repack -q -d
  git -C "${dir}" fetch -q --deepen=1
  echo "${label} (since=${since}):"
  for c in $(git -C "${dir}" rev-list --all); do
    local subject first_parent
    subject="$(git -C "${dir}" log -1 --format=%s "$c" | cut -d: -f1)"
    first_parent="$(git -C origin.git rev-list --parents -n1 "$c" | cut -s -d' ' -f2)"
    if [[ -n "${first_parent}" ]] && ! git -C "${dir}" cat-file -e "${first_parent}" 2>/dev/null; then
      echo "  ${subject}: SKIPPED (first parent not in local ODB) -> not stored"
    else
      echo "  ${subject}: stored"
    fi
  done
}

git init -q --bare -b main origin.git
tree="$(git -C origin.git hash-object -t tree -w --stdin </dev/null)"

# Run 0 (previous incremental run) started at 2024-01-02T12:00:00Z; R and F were collected then.
R=$(commit "2024-01-01T00:00:00Z" "R: base on main")
F=$(commit "2024-01-02T10:00:00Z" "F: feature branch head, pushed before run 0" "$R")
# After run 0: two changes land on main.
Q=$(commit "2024-01-02T13:00:00Z" "Q: main commit after run 0" "$R")
P=$(commit "2024-01-02T14:00:00Z" "P: main commit after run 0 (deployed)" "$Q")
M=$(commit "2024-01-02T16:00:00Z" "M: merge request merges feature into main" "$P" "$F")
git -C origin.git update-ref refs/heads/main "$M"
git -C origin.git update-ref refs/heads/feature "$F"

echo "git $(git --version | cut -d' ' -f3)"
echo "expected new commits for run 1: Q, P, M"
incremental_run "2024-01-02T12:00:00Z" run1

# Run 2 (next day): since = start of run 1; a new commit lands on main.
N=$(commit "2024-01-03T09:00:00Z" "N: next main commit" "$M")
git -C origin.git update-ref refs/heads/main "$N"
incremental_run "2024-01-02T18:00:00Z" run2
echo "=> Q and P were never stored by any run."

Output (the same with git 2.39.5, the version in the apache/devlake:v1.0.3-beta15 image, and with git 2.56.0):

expected new commits for run 1: Q, P, M
run1 (since=2024-01-02T12:00:00Z):
  M: stored
  P: SKIPPED (first parent not in local ODB) -> not stored
  F: stored
  R: stored
run2 (since=2024-01-02T18:00:00Z):
  N: stored
  M: SKIPPED (first parent not in local ODB) -> not stored
  F: stored
  R: stored
=> Q and P were never stored by any run.

In run 2, M is skipped too, but that is harmless because M was stored in run 1. This is the case #7720 was designed for.

The script replays DevLake's git commands and skip rule; we have not run this exact history through a fresh DevLake instance. Our production data matches its prediction: the commits the script says are lost are the ones missing from our database. To reproduce end to end:

1. Push R, create branch feature with F, and run a blueprint once.
2. Land Q and then P on the default branch, and merge feature with a merge commit (M).
3. Run the blueprint again (incremental).
4. Check commits / repo_commits: Q and P should be missing, while commit_parents has (M, P).

Anything else

• Frequency: every incremental run where the trigger condition happens. The gaps pile up silently.
• Workaround: a full sync. Per [Question][GitLab] GitLab data collection is incomplete #8818, "Full Refresh" collects the missing commits.
• Proposed fix (I plan to open a PR): after --deepen=1, keep running git fetch --deepen=1 while any shallow boundary commit is still newer than since, with an iteration cap. When the loop ends, every boundary commit is older than since, so it was stored by an earlier run, and the existing skip rule becomes harmless again. The first full sync with timeAfter uses the same path and benefits too. In the script above this stores Q, P and M in run 1, with both git versions. Already-lost commits still need one full sync after upgrading.
• Environment: USE_GO_GIT_IN_GIT_EXTRACTOR=false (libgit2), SkipCommitStat=false, SkipCommitFiles=true, NoShallowClone=false.
• The fetch sequence and the skip rule are unchanged in v1.0.3-beta18 and on main (e355317df).

Version

v1.0.3-beta15

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct