Comparing changes

Reproduce the nightly QA pytorch_inference OOM without Elasticsearch: a new dev-tools driver drives a sustained inference load through the standalone process (ELSER model, no ES) while sampling RSS, collapsing the ~6h QA cycle into a direct memory trace in minutes. A new run_rss_probe CI action builds only the linux x86_64 distribution and runs the probe on the 64 GiB build agent, skipping the ES/QA suites. The action and RSS_PROBE_* overrides are threaded through the pytorch-builds trigger (pr-builds rejects direct API builds). Co-authored-by: Cursor <cursoragent@cursor.com>

…pro (#3226) Add a prebuilt-distribution mode to the RSS probe so we can reproduce the pytorch_inference OOM against an exact, already-built ml-cpp+libtorch combo without any rebuild. When RSS_PROBE_DIST_BUILD is set, the pipeline skips the libtorch Docker image build and the ml-cpp compile, and the probe step pulls the linux-x86_64 distribution from that build (cross-build artifact download) before running the standalone load. Enables the clean single-variable test b0c52aa + bb63328c by reusing A/B Arm B's distribution. Co-authored-by: Cursor <cursoragent@cursor.com>

The probe step resolved the downloaded distribution zip via a relative `find build/distributions` and unzipped it through a `../` prefix, both of which depended on the command's working directory matching the artifact download location. On the prebuilt path this left DIST_ZIP empty and the unzip failed with status 9. Download into an absolute dir, locate the zip and binary by absolute path, unzip with -d, and fail loudly if either is missing. Co-authored-by: Cursor <cursoragent@cursor.com>

…line-upload interpolation (#3226) buildkite-agent pipeline upload interpolates ${VAR} in the step command at upload time, which blanked out $PWD, $(find ...) and the probe knobs that are meant to be evaluated at runtime (the step failed with empty paths and mkdir ''). Move the whole probe body into dev-tools/run_rss_probe_ci.sh and drive it purely through the step env: block, so the YAML command is a single literal invocation with no shell variables for Buildkite to interpolate. Co-authored-by: Cursor <cursoragent@cursor.com>

…hase the OOM (#3226) The fixed 512-token single-shape workload left RSS flat at ~1 GiB, far from the 34-40 GiB QA OOM. Add --vary-tokens to randomise each request's sequence length over [--min-tokens, --num-tokens], mimicking a real variable-length corpus (wikipedia) and exercising many distinct tensor shapes, which is what grows the caching allocator. Raise the CI defaults accordingly: 20000 requests, 8 allocations, variable 1-512 token lengths. Co-authored-by: Cursor <cursoragent@cursor.com>

#3226) The QA OOM runs the default .elser-2-elasticsearch endpoint (adaptive allocations, min 0 / max 32, threads_per_allocation 1) over wikipedia-1g, whose documents skew toward the 512-token cap. Add --token-skew long (Beta distribution biased to the maximum) so the synthetic load better matches the real length distribution, and thread it through the CI env passthrough. Co-authored-by: Cursor <cursoragent@cursor.com>

threading.Thread uses a private _stop() method internally; naming the watcher's Event attribute self._stop shadowed it, so Thread.join() -> _wait_for_tstate_lock -> self._stop() raised "'Event' object is not callable" at cleanup, failing the step and suppressing the verdict even though the probe ran fine. Rename the attribute to _stop_event. Co-authored-by: Cursor <cursoragent@cursor.com>

Navigation Menu

Repository navigation

Comparing changes

Open a pull request

Uh oh!

Commits on Oct 2, 2026

This comparison is taking too long to generate.

Uh oh!