fix(bench): resolve the #361 harness's source SHA from soup_cli's tree, not the checkout's HEAD - #1382

@drakeo338, thank you. This does what #1327 asked, and it does it the way the repository already does it elsewhere. Three things stand out:

• Both harnesses now answer "which tree ran?" the same way. Your _source_sha() is the #1085 function from variant2_gate.py, and the row format stays <sha>, <sha>-dirty or unknown, so rows that are already published stay comparable.
• The test pins the part that matters. It answers --show-toplevel with a directory that belongs neither to the package nor to the harness, and it asserts the cwd of every git call. Any way of reading the commit from the wrong tree therefore fails it. Against main's harness both cases fail on the value itself (2 failed, 31 passed).
• You widened the shape check instead of dropping it. [0-9a-f]{40}(-dirty)? at tests/test_issue361_nf4_throughput.py:178 keeps the real-git test meaningful on a dirty CI checkout.

From my side this is ready. Nothing below holds the merge. It now waits only on the 13 required checks, which are queued on 5d86dcc1.

What I ran

Everything ran on CPU with the GPU hidden, on 5d86dcc1, which already sits on today's main (1e8a0ae5).

Tests. tests/test_issue361_nf4_throughput.py: 33 passed. Your two skips are the tests that importorskip("torch"), and they run here. Adding tests/test_issue379_variant2_gate.py and the Qwen4 benchmarks-index test gives 67 passed. ruff check src/soup_cli/ tests/ scripts/ benchmarks/ passes on this head (ruff 0.16.6).

_source_sha() with real git, with the harness loaded through importlib:

Ten mutations of _source_sha() in benchmarks/harness/issue361_nf4_throughput.py, each run against the #361 test file:

Every mutation that moves the commit to the wrong tree or drops -dirty fails a test. I would not test the last two guards. Git 2.55 exits 128 rather than printing an empty toplevel, even in a bare repository, and a regular package always has a __file__.

Optional follow-ups (none of them holds the merge)

1. Three error branches have no test. Narrowing the except to either half, or deleting the 40-hex check, passes all 33 tests. The git archive case is the one that happens in practice: git rev-parse --show-toplevel exits 128 there, and with check=True that is a CalledProcessError.

These three methods fit into TestTheRowNamesItsTree as they are, because the file already imports subprocess, SimpleNamespace, Any, Path and pytest. They pass on 5d86dcc1, and each one fails exactly one of those three mutations (36 passed unmutated; 1 failed under each):

    def test_source_sha_is_unknown_outside_a_git_checkout(
        self, monkeypatch: pytest.MonkeyPatch
    ) -> None:
        """A ``git archive`` extract has no ``.git``, so ``git rev-parse
        --show-toplevel`` exits 128, which ``check=True`` raises as
        ``CalledProcessError``."""

        def fake_run(command: list[str], *, cwd: Path, **kwargs: object) -> Any:
            raise subprocess.CalledProcessError(128, command)

        monkeypatch.setattr(subprocess, "run", fake_run)
        assert _harness._source_sha() == "unknown"

    def test_source_sha_is_unknown_without_a_git_binary(
        self, monkeypatch: pytest.MonkeyPatch
    ) -> None:
        def fake_run(command: list[str], *, cwd: Path, **kwargs: object) -> Any:
            raise FileNotFoundError(2, "No such file or directory", command[0])

        monkeypatch.setattr(subprocess, "run", fake_run)
        assert _harness._source_sha() == "unknown"

    def test_source_sha_does_not_record_a_head_that_is_not_a_commit(
        self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
    ) -> None:
        def fake_run(command: list[str], *, cwd: Path, **kwargs: object) -> Any:
            if command == ["git", "rev-parse", "--show-toplevel"]:
                return SimpleNamespace(stdout=f"{tmp_path}\n")
            if command == ["git", "rev-parse", "HEAD"]:
                return SimpleNamespace(stdout="not-a-commit\n")
            return SimpleNamespace(stdout="")

        monkeypatch.setattr(subprocess, "run", fake_run)
        assert _harness._source_sha() == "unknown"

2. -dirty also fires on this harness's own outputs. git status --untracked-files=normal counts untracked files anywhere in the checkout. The docstring's example (:33-36) writes --shards ./shards-llama8b-nf4 and --json ./rows/issue361-post-repair.json into the current directory, and neither path is ignored. Run from the repository root, the first run is still clean, because _versions() (:391) runs before sharding (:429) and before the first JSON write (:522). Every re-run that reuses the shards, though, is stamped -dirty. I checked both paths with real git. This errs in the safe direction, variant2_gate.py behaves the same way, and #1327 asked for exactly this status call, so no change is needed. If you want to avoid it, point the example's paths outside the checkout, or scope the call with -- src benchmarks/harness, which still catches a local edit to soup_cli or to the harness itself.

3. One case the #1085 function does not cover. This PR inherits it; it did not introduce it. The case is a wheel installed into a gitignored .venv/ inside the checkout. I copied the installed 0.75.1 wheel into <checkout>/.venv/Lib/site-packages and imported it from there. --show-toplevel finds the checkout, .venv/ is ignored so git status is empty, and the row gets the checkout's clean commit (5d86dcc1…) for code that differs from src/ in 187 files. variant2_gate.py reports the same commit. Running git ls-files --error-unmatch -- <soup_cli/__init__.py> in that root tells the two apart: it exits 1 for the wheel's copy and 0 for src/soup_cli/__init__.py. The fix belongs in every copy of the function, so it fits better with item 4 than in this PR.

4. One shared helper. #1327's fix path also offered a small helper that both harnesses import, like bitexact.py, with a test that both use the same function object. This PR's _source_sha() is byte-identical to variant2_gate.py:213-255, and #1258 adds a third copy that is already scoped differently (-- src), so the copies have started to drift. Recording soup_cli.__file__ in the versions blob, the other optional item in #1327, would also make a case like item 3 visible in the row itself. cuda_graph_decode_bench.py already records it as soup_import_path.