fix(kubernetes): serialize lifecycle cleanup with sandbox restart - #4078

Summary

A Kubernetes reconciliation pass can retain a stopped or stopping Sandbox LIST snapshot while restart creates a replacement supervisor Pod at the same name. Cleanup can then delete the replacement before a later CR update detects the conflict. Serialize lifecycle mutations and reconciliation per sandbox within one driver instance, and refresh the CR under the gate before deciding whether to clean up.

This is a proposed quick fix for a reproduced race. The historical sandbox-stop-start CI failure lacked Kubernetes Pod diagnostics, so its exact cause remains unproven.

Related Issue

Refs #3954. No new issue was created, as explicitly requested; this PR does not close the broader investigation. This cleanup guard complements the bootstrap fence publication work in #4067 and diagnostics in #4072.

Changes

• Share per-sandbox mutation gates across driver clones for create, stop, start, delete, and reconciliation. Skip busy sandboxes until the next reconciliation pass while preserving concurrency across sandboxes.
• Refresh each listed CR under the gate and reject a replaced CR UID or changed sandbox identity before examining companion resources.
• Add four regression tests covering stale stopped/releasing/suspending snapshots on both Sandbox API versions, a busy restart, CR replacement, and lifecycle gate sharing across clones. Update the driver README and cluster debugging skill.

The gate is process-local: separate gateway or external driver processes can still race. This draft does not change generic Kubernetes 404 error mapping or replace generation/UID fencing with distributed exclusion.

Testing

Rebased without conflicts onto 76cfd0e31d5e1633db7ccd86ad9023ef7a2461b2 (includes #4076). Current candidate: 9db51c01936c6eeafdc59c74612e9839e1623e1d. git range-diff confirms this PR's patch is unchanged.

• Fresh Kubernetes driver library run on the rebased candidate: all 274 tests pass, including the four cleanup regressions.
• Fresh cargo clippy --offline -p openshell-driver-kubernetes --all-targets -- -D warnings and git diff --check pass.
• Full enabled Branch E2E run 36933371807, attempt 1 passed, including both Kubernetes Agent Sandbox API smoke lanes and ubuntu-k3s conformance.
• Three actual successful executions of affected stop/start coverage per target lane.

PR head, workflow heads, binary artifact sources, preparation checkouts, and image tags were verified against the candidate SHA. Carried-forward GitHub results with regenerated job IDs are excluded from repeat counts using actual execution timestamps and logs. No failures were observed in this campaign.

Optional Kubernetes HA and credential-driver suites were disabled in the full run and are not counted as passing coverage. Passing K3s logs show actual conformance archive execution and its success assertion; the pinned archive includes the unignored lifecycle scenario, but passing per-command output is suppressed.

The original commit passed the full configured pre-commit hook. Local cluster E2E previously stopped before scenarios because Docker was unavailable; that preflight attempt is not qualification evidence for this candidate.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Relevant architecture and debugging documentation updated