fix(e2e): fail loudly when OpenShift detection is inconclusive - #4090

Summary

The Kubernetes e2e wrapper decided whether the target cluster was OpenShift with a probe that could fail silently, leaving OPENSHIFT_DETECTED=0 and reconfiguring the entire run down the vanilla-Kubernetes path. Detection now distinguishes a conclusive "not OpenShift" answer from a discovery failure, retries the latter, and aborts rather than guessing.

Related Issue

Closes #4088

Changes

The old probe was:

if kctl api-resources --api-group=route.openshift.io --no-headers 2>/dev/null | grep -q .; then

Three properties made failure silent: kubectl's exit status was discarded because it was the left side of a pipe, its stderr went to /dev/null, and the probe ran exactly once. api-resources is a discovery call that fans out to every aggregated APIService, so it returns partial results or fails outright during transient unavailability, throttling, or an auth blip — none of which mean the cluster is not OpenShift.

• Probe kubectl get --raw /apis/route.openshift.io/v1 instead. It asks about one API group and does not depend on full aggregated discovery.
• Capture kubectl's exit status directly rather than masking it behind a pipeline, and keep its stderr so a failure can be reported.
• Treat only a NotFound answer as conclusive evidence the cluster is not OpenShift. Retry anything else with exponential backoff (5 attempts, starting at 2s).
• On exhausted retries, exit non-zero naming the probe, the exit status, and the underlying error, instead of continuing with OPENSHIFT_DETECTED=0.
• Log the outcome in both directions, so "not OpenShift" is distinguishable from "the probe never ran". Previously only the OpenShift branch printed anything.
• Add OPENSHELL_E2E_OPENSHIFT=1|0 to force the answer and skip the probe, following the existing OPENSHELL_E2E_* convention. An unrecognized value is a hard error rather than a silent default.

The logic lives in e2e/support/gateway-common.sh so it is sourceable and unit-testable with a fake kubectl on PATH, following the existing test-e2e-image-overrides.sh precedent. The oc-is-required check is preserved, moved into the branch that follows detection.

Why this matters

OPENSHIFT_DETECTED gates seven behaviors, including the SCC values overlay, the privileged SCC grant for the openshell-sandbox service account, and using a passthrough Route instead of a port-forward as the gateway transport. Misdetection did not produce a detection error — it produced security contexts that SCC admission rejects and sandbox connect attempting SSH over a port-forward, which the script's own header comment explains can never complete. The failures surfaced far from the cause and read as product bugs.

Testing

• Checks appropriate to the affected code and behavior pass
• Unit tests added/updated (if applicable)
• E2E tests added/updated (if applicable)

mise run pre-commit passes. shellcheck -x is clean on all three changed shell files, and bash -n e2e/with-kube-gateway.sh is clean.

New tasks/scripts/test-e2e-openshift-detection.sh, registered as test:e2e-openshift-detection and added to the [test] depends list. It puts a fake kubectl ahead of PATH with an invocation counter, so it can assert how many times the probe ran:

1. OpenShift cluster → 1, one probe call
2. Conclusive NotFound → 0, exactly one call (no pointless retry), and the not-OpenShift line is logged
3. Transient failure twice then success → 1, three calls
4. Persistent failure → non-zero exit, result is not 0, stderr names the probe path and the underlying error
5. OPENSHELL_E2E_OPENSHIFT=1 against a broken kubectl → 1, zero probe calls
6. OPENSHELL_E2E_OPENSHIFT=false against an OpenShift kubectl → 0, zero probe calls
7. OPENSHELL_E2E_OPENSHIFT=maybe → non-zero exit with an explanatory message

The suite was mutation-tested to confirm it has teeth: reintroducing the original bug (treating every failure as conclusive absence) and separately removing the retry loop each make case 3 fail.

Verified against a real OpenShift cluster

The one judgement call here is that conclusive-absence is recognized from kubectl's error text, since get --raw returns exit 1 for every failure class. That was checked against a live OpenShift 4.x cluster (RHCOS 9.6) by calling the helper directly:

Real kubectl emits Error from server (NotFound): the server could not find the requested resource for an absent group, which satisfies both matchers. Note the third row: on persistent failure the result is empty rather than 0, so even if that wording changed in a future kubectl, the failure mode is a loud abort and never the silent-vanilla regression this issue is about.

Not verified: a full with-kube-gateway.sh run end to end on OpenShift. The detection function itself was exercised against the live cluster as above.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)