A patch release for workers that run memory-hungry tests and for CAS deployments on Azure under heavy load. Workers now measure and kill an action across its whole process tree, so a test that Bazel starts in its own process group is sized and enforced by its real memory. The CAS no longer stalls every upload behind uploads whose clients disconnected, and an Azure store can cap the requests it keeps in flight so a burst of existence checks cannot exhaust the host's local ports. The release also fixes five OSV advisories.

Two new settings are opt-in and off by default. One behavior changes on upgrade with no configuration change: memory readings for tests get larger (see Upgrade notes).

What's Changed

Tests sized by the wrapper instead of the test

Problem: Bazel's test-setup.sh starts the real test in its own process group (set -m). The worker measured only the action's original group, so it recorded the wrapper's few MB instead of the test's peak. Historical hints sized such tests far too small, memory enforcement never fired, and kills reached only the wrapper, leaving the test running as an orphan. A test that outgrew its worker took the pod down, and the scheduler retried it at the same size.

Summary: The worker now samples and kills every process group the action owns. Timeout, cancel and memory kills reach the real test. An opt-in memory pressure kill ends the action furthest over its reservation when the worker's cgroup thrashes, and never one that already holds the largest reservation the worker advertises. Origin events now record why each attempt ended, so retries caused by lost workers or memory kills are visible.

PRs addressed:

• Sample and kill an action across its whole process tree, and kill the action furthest over its reservation under worker memory pressure by @amankrx in (#3038) - (bed7314)
• Report action phases, transfer bytes and why an attempt ended in origin events by @amankrx in (#3040) - (19cdbe7)

CAS uploads stalling behind abandoned uploads

Problem: A client that sent an upload's last byte and then disconnected left the upload parked for a resume. Its commit kept its place in the filesystem store's write queue, but nothing polled it, so it took a write permit when its turn came and held it. Once parked uploads held every permit, every write and every cache fill on that CAS waited about a minute while the CPU sat idle, and the queue refilled faster than the idle sweep cleared it.

Summary: Once the last byte has arrived, the commit runs on its own task and releases its permit as soon as it finishes; the parked entry only holds the result for a client that resumes. Background commits are bounded at 15 minutes.

PRs addressed:

• Finish an upload's commit on its own task when the client leaves after its last byte, instead of parking it unpolled in the store's write queue by @amankrx in (#3044) - (779c3e7)

Azure connections exhausting local ports

Problem: Azure Blob Storage speaks HTTP/1.1, so every request in flight holds its own connection and local port. Nothing limited how many requests an Azure store sent at once: an existence check for a large batch sent a HEAD per digest, and large uploads staged their blocks the same way. A busy CAS could run out of local ports and fail with AddrNotAvailable.

Summary: A new max_concurrent_requests setting caps the requests each Azure store keeps in flight, uploads included. At the cap, requests wait in the process instead of opening connections. Requests have connect and read timeouts, and the store warns, at most once a minute, when the host runs out of local ports.

PRs addressed:

• Bound the requests an Azure store keeps in flight, uploads included by @MarcusSorealheis in (#2985) - (4a6f505)
• Express the Azure port-hint interval in minutes by @MarcusSorealheis in (#3050) - (315f096)

Missing and mis-sized blobs in layered stores

Problem: An existence cache kept reporting a blob as present after it left the object store, so a client told to re-upload a missing input was told it already existed, and a CAS that forwards Execute to a separate scheduler dropped the detail naming the missing blob. A lazy fill into a compression fast tier sized its block index from an unknown upload size and could abort the process. Verify's repair could delete the durable filesystem copy.

Summary: The existence cache drops a blob as soon as a read or an action finds it missing, and the missing-blob detail survives a forwarded status, so the client re-uploads and retries. The compression store grows its block index as blocks are written. Repair keeps the durable copy.

PRs addressed:

• Stop the existence cache vouching for a blob once it is found missing, and keep the missing-blob detail through a forwarded status by @amankrx in (#3047) - (4ba0f7d)
• Grow the compression store's block index as blocks are written instead of sizing it from the upload size by @amankrx in (#3045) - (722ce48)
• Verify's repair no longer deletes the durable filesystem copy by @ashishsonamm in (#3046) - (19e8462)

Execution and toolchains

Problem: Binary headers were dropped by forward_headers, a namespaced action saw a /proc that did not match its PID namespace, and remote lre-rs actions on macOS failed because the toolchain did not declare the rustc dylibs and rust-objcopy as inputs.

Summary: Each is fixed at its source.

PRs addressed:

• Forward binary headers in forward_headers by @jtszalay in (#3017) - (e76fd95)
• Give a namespaced action a /proc that matches its PID namespace by @b7r6 in (#3024) - (db200db)
• Declare macOS rustc dylibs and rust-objcopy as lre-rs toolchain inputs by @modernmedici in (#3036) - (f01785c)

Security

Problem: Five dependencies had OSV advisories, three of them in the server binary.

Summary: tokio moves to 1.53.3, ctutils to 0.4.4 and webpki-root-certs to 1.0.10, and MSAL Node is pinned in the Azurite test runner.

PRs addressed:

• Update five dependencies with OSV advisories by @MarcusSorealheis in (#3035) - (42b4b4d)
• Override MSAL Node to 7.1.0 in the Azurite test runner by @MarcusSorealheis in (#3048) - (7e55aeb)

CI and docs

Problem: Sanitizer and fuzzing coverage did not run regularly, coverage publishing failed when the cache was down, and the nightly Clippy rejected an assertion in the scheduler tests.

Summary: AddressSanitizer and fuzzing run nightly and on release pull requests, coverage publishes independently of cache uploads, the assertion uses assert_eq!, and the README's use cases move into AGENTS.md.

PRs addressed:

• Run AddressSanitizer and fuzzing nightly by @matdexir in (#3029) - (e4b5314)
• Keep coverage reports publishing during cache outages by @MarcusSorealheis in (#3033) - (0c4a6cf)
• Compare the swept attempts with assert_eq in the executing sweep test by @MarcusSorealheis in (#3053) - (0909e73)
• Move README use cases into AGENTS.md as an agent index by @modernmedici in (#3043) - (3a63e8e)

Upgrade notes

• Larger memory readings. Tests that run in their own process group now report their real peak, so historical hints and resource_enforcement see larger numbers after the upgrade. With memory enforcement on, actions that were over their reservation but unmeasured are now killed and retried at a larger size.
• New opt-in settings, off by default: max_concurrent_requests on Azure stores, and memory_pressure_kill_percent / memory_pressure_kill_after_s under resource_enforcement (requires memory: soft). Earlier versions reject these as unknown fields, so set them only on 1.8.2 or later. When setting max_concurrent_requests, keep max_concurrent_reads below it: a download holds its request permit while its body streams.
• No other configuration changes are needed.

Known issue

• A resumed ByteStream write whose RPC is cancelled after its background commit finished can re-spawn the finished commit and panic that task. The CAS keeps running and the blob is stored; the next resume of that upload gets an Internal error and the client retries.

New Contributors

• @jtszalay made their first contribution in #3017

Full Changelog: v1.8.1...v1.8.2