Skip to content

Commit 05a096a

Browse files
pelikhanCopilot
pelikhan
and
Copilot
authored
Add least-privilege failure reporting and systematic scenario matrix to Windows CLI integration (#67410)
* Add least-privilege failure reporting and systematic scenario matrix to Windows CLI integration - conclusion job now creates (or comments on an existing open) tracking issue naming failed jobs and linking the run when build/integration fails, scoped to issues:write on this job only. - scenarioMatrix is generated from declared shell/launch-mode/env-shape/ path-style dimension arrays instead of a hand-picked list, with bounded growth (dimension-pair coverage) to avoid blowing the job timeout. - per-scenario failures are now tagged with the scenario name for faster triage. Fixes #66822 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix duplicate scenario in Windows CLI dimension-pair generation Deterministic execution of the scenario-generation logic (via PowerShell AST parser + actual loop execution in a container, not just brace counting) caught a genuine duplicate: dimension pair 1 already produces $representativeShell/path/default/workspace-first (its "path" launch mode always resolves to the workspace-first PATH style), but dimension pair 2's skip condition only excluded env=default+path=default, not env=default+path=workspace-first. Fixed the skip condition so the 33 generated scenarios are now verified unique, with full shell x launch and env x path coverage confirmed by execution. Fixes #66822 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix PATH-resolution mismatch in env/path scenario generation copilot-pull-request-reviewer flagged that dimension pair 2's generated scenarios paired every path style (including "default") with launch="path". For path="default" the PATH-rearrangement switch is a no-op, leaving PATH unmodified, so the workspace copy of gh-aw.exe is never resolvable via `where gh-aw` on a clean runner -- those scenarios would fail before invoking the CLI at all. Fix: path="default" now pairs with launch="direct" (consistent with dimension pair 1's own default/default scenario, since PATH rearrangement has no effect on a direct invocation anyway); every other path style still pairs with launch="path". Updated the duplicate-skip condition accordingly (now skips both combinations dimension pair 1 already covers for the representative shell), yielding 32 unique scenarios (down from 33, since one more overlap is now correctly excluded). Verified via actual PowerShell execution in a container (not just static review): scenario count, name uniqueness, full shell x launch and env x path coverage, and a new explicit regression check that no launch="path" scenario has path="default". Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
1 parent 1976c94 commit 05a096a

1 file changed

Lines changed: 121 additions & 57 deletions

File tree

‎.github/workflows/windows-cli-integration.yml‎

Lines changed: 121 additions & 57 deletions
Original file line numberDiff line numberDiff line change
@@ -178,28 +178,50 @@ jobs:
178178
Copy-Item -Path $binary -Destination (Join-Path $parenDir "gh-aw.exe") -Force
179179
Copy-Item -Path $binary -Destination (Join-Path $toolcacheDir "gh-aw.exe") -Force
180180
181-
$scenarioMatrix = @(
182-
@{ name = "pwsh-default/direct/default/default"; shell = "pwsh-default"; launch = "direct"; env = "default"; path = "default" },
183-
@{ name = "pwsh-noprofile/direct/default/default"; shell = "pwsh-noprofile"; launch = "direct"; env = "default"; path = "default" },
184-
@{ name = "powershell-default/direct/default/default"; shell = "powershell-default"; launch = "direct"; env = "default"; path = "default" },
185-
@{ name = "powershell-noprofile/direct/default/default"; shell = "powershell-noprofile"; launch = "direct"; env = "default"; path = "default" },
186-
@{ name = "cmd/direct/default/default"; shell = "cmd"; launch = "direct"; env = "default"; path = "default" },
187-
@{ name = "pwsh-default/path/default/workspace-first"; shell = "pwsh-default"; launch = "path"; env = "default"; path = "workspace-first" },
188-
@{ name = "pwsh-noprofile/path/default/workspace-first"; shell = "pwsh-noprofile"; launch = "path"; env = "default"; path = "workspace-first" },
189-
@{ name = "powershell-default/path/default/workspace-first"; shell = "powershell-default"; launch = "path"; env = "default"; path = "workspace-first" },
190-
@{ name = "powershell-noprofile/path/default/workspace-first"; shell = "powershell-noprofile"; launch = "path"; env = "default"; path = "workspace-first" },
191-
@{ name = "cmd/path/default/toolcache-first"; shell = "cmd"; launch = "path"; env = "default"; path = "toolcache-first" },
192-
@{ name = "pwsh-default/path/default/toolcache-first"; shell = "pwsh-default"; launch = "path"; env = "default"; path = "toolcache-first" },
193-
@{ name = "pwsh-noprofile/path/default/toolcache-first"; shell = "pwsh-noprofile"; launch = "path"; env = "default"; path = "toolcache-first" },
194-
@{ name = "powershell-default/path/default/toolcache-first"; shell = "powershell-default"; launch = "path"; env = "default"; path = "toolcache-first" },
195-
@{ name = "powershell-noprofile/path/default/toolcache-first"; shell = "powershell-noprofile"; launch = "path"; env = "default"; path = "toolcache-first" },
196-
@{ name = "pwsh-default/path/default/duplicate-workspace"; shell = "pwsh-default"; launch = "path"; env = "default"; path = "duplicate-workspace" },
197-
@{ name = "pwsh-default/path/no-color/workspace-first"; shell = "pwsh-default"; launch = "path"; env = "no-color"; path = "workspace-first" },
198-
@{ name = "pwsh-default/path/term-dumb/workspace-first"; shell = "pwsh-default"; launch = "path"; env = "term-dumb"; path = "workspace-first" },
199-
@{ name = "pwsh-default/path/term-unset/workspace-first"; shell = "pwsh-default"; launch = "path"; env = "term-unset"; path = "workspace-first" },
200-
@{ name = "pwsh-default/path/ci-true/workspace-first"; shell = "pwsh-default"; launch = "path"; env = "ci-true"; path = "workspace-first" },
201-
@{ name = "pwsh-default/path/ci-unset/workspace-first"; shell = "pwsh-default"; launch = "path"; env = "ci-unset"; path = "workspace-first" }
202-
)
181+
# Declared coverage dimensions. The matrix below is generated from these
182+
# arrays (not hand-picked) so every shell is exercised directly and via
183+
# PATH resolution, and every env-shape/path-style pair is exercised at
184+
# least once. A full cross product (shells x launch x env x path) would
185+
# be 240+ scenarios and risk the integration job's timeout, so growth is
186+
# additive across dimension pairs rather than multiplicative across all
187+
# four: shell coverage is paired with launch-mode, and env-shape coverage
188+
# is paired with path-style via one representative shell/launch.
189+
$shells = @("pwsh-default", "pwsh-noprofile", "powershell-default", "powershell-noprofile", "cmd")
190+
$launchModes = @("direct", "path")
191+
$envShapes = @("default", "no-color", "term-dumb", "term-unset", "ci-true", "ci-unset")
192+
$pathStyles = @("default", "workspace-first", "toolcache-first", "duplicate-workspace")
193+
$representativeShell = "pwsh-noprofile"
194+
195+
$scenarioMatrix = [System.Collections.Generic.List[hashtable]]::new()
196+
197+
# Dimension pair 1: shell x launch-mode, every shell exercised both as a
198+
# direct invocation and resolved via PATH.
199+
foreach ($shell in $shells) {
200+
foreach ($launch in $launchModes) {
201+
$path = if ($launch -eq "path") { "workspace-first" } else { "default" }
202+
$scenarioMatrix.Add(@{ name = "$shell/$launch/default/$path"; shell = $shell; launch = $launch; env = "default"; path = $path })
203+
}
204+
}
205+
206+
# Dimension pair 2: env-shape x path-style, every environment variant
207+
# exercised across every PATH arrangement via a fixed representative
208+
# shell. The "default" path style means "do not resolve via PATH at
209+
# all", so it is paired with direct invocation (matching dimension
210+
# pair 1's own default/default direct scenario); every other path
211+
# style is paired with PATH-based launch, since rearranging PATH has
212+
# no effect on a direct invocation. Dimension pair 1 already
213+
# generates $representativeShell/direct/default/default and
214+
# $representativeShell/path/default/workspace-first, so both are
215+
# skipped here to avoid duplicates.
216+
foreach ($env in $envShapes) {
217+
foreach ($path in $pathStyles) {
218+
if ($env -eq "default" -and ($path -eq "default" -or $path -eq "workspace-first")) { continue }
219+
$launch = if ($path -eq "default") { "direct" } else { "path" }
220+
$scenarioMatrix.Add(@{ name = "$representativeShell/$launch/$env/$path"; shell = $representativeShell; launch = $launch; env = $env; path = $path })
221+
}
222+
}
223+
224+
Add-Content $env:GITHUB_STEP_SUMMARY "Generated $($scenarioMatrix.Count) scenarios from $($shells.Count) shells x $($launchModes.Count) launch modes x $($envShapes.Count) env shapes x $($pathStyles.Count) path styles"
203225
204226
foreach ($scenario in $scenarioMatrix) {
205227
$env:PATH = $originalPath
@@ -222,45 +244,49 @@ jobs:
222244
default {}
223245
}
224246
225-
if ($scenario.launch -eq "path") {
226-
$whereResult = Invoke-CliProcess -FilePath "cmd" -ArgumentList @("/d", "/s", "/c", "where gh-aw") -EnvironmentOverrides $envOverrides -UnsetEnvironment $unsetVars
227-
$firstResolved = ($whereResult.Output -split "`r?`n" | Where-Object { $_ -match "gh-aw\.exe$" } | Select-Object -First 1)
228-
$expectedPathDirectory = if ($scenario.path -eq "toolcache-first") { $toolcacheDir } else { $workspace }
229-
$expectedExecutable = Join-Path $expectedPathDirectory "gh-aw.exe"
230-
if (-not [System.String]::Equals($firstResolved, $expectedExecutable, [System.StringComparison]::OrdinalIgnoreCase)) {
231-
throw "PATH resolution mismatch for '$($scenario.name)': $firstResolved"
247+
try {
248+
if ($scenario.launch -eq "path") {
249+
$whereResult = Invoke-CliProcess -FilePath "cmd" -ArgumentList @("/d", "/s", "/c", "where gh-aw") -EnvironmentOverrides $envOverrides -UnsetEnvironment $unsetVars
250+
$firstResolved = ($whereResult.Output -split "`r?`n" | Where-Object { $_ -match "gh-aw\.exe$" } | Select-Object -First 1)
251+
$expectedPathDirectory = if ($scenario.path -eq "toolcache-first") { $toolcacheDir } else { $workspace }
252+
$expectedExecutable = Join-Path $expectedPathDirectory "gh-aw.exe"
253+
if (-not [System.String]::Equals($firstResolved, $expectedExecutable, [System.StringComparison]::OrdinalIgnoreCase)) {
254+
throw "PATH resolution mismatch: $firstResolved"
255+
}
232256
}
233-
}
234257
235-
switch ($scenario.shell) {
236-
"pwsh-default" {
237-
$cmd = if ($scenario.launch -eq "path") { "& gh-aw --help" } else { "& '$binary' --help" }
238-
Invoke-CliProcess -FilePath "pwsh" -ArgumentList @("-Command", $cmd) -EnvironmentOverrides $envOverrides -UnsetEnvironment $unsetVars -AssertNoAnsi:$assertNoAnsi | Out-Null
239-
}
240-
"pwsh-noprofile" {
241-
$cmd = if ($scenario.launch -eq "path") { "& gh-aw --help" } else { "& '$binary' --help" }
242-
Invoke-CliProcess -FilePath "pwsh" -ArgumentList @("-NoProfile", "-Command", $cmd) -EnvironmentOverrides $envOverrides -UnsetEnvironment $unsetVars -AssertNoAnsi:$assertNoAnsi | Out-Null
243-
}
244-
"powershell-default" {
245-
$cmd = if ($scenario.launch -eq "path") { "& gh-aw --help" } else { "& '$binary' --help" }
246-
Invoke-CliProcess -FilePath "powershell" -ArgumentList @("-Command", $cmd) -EnvironmentOverrides $envOverrides -UnsetEnvironment $unsetVars -AssertNoAnsi:$assertNoAnsi | Out-Null
247-
}
248-
"powershell-noprofile" {
249-
$cmd = if ($scenario.launch -eq "path") { "& gh-aw --help" } else { "& '$binary' --help" }
250-
Invoke-CliProcess -FilePath "powershell" -ArgumentList @("-NoProfile", "-Command", $cmd) -EnvironmentOverrides $envOverrides -UnsetEnvironment $unsetVars -AssertNoAnsi:$assertNoAnsi | Out-Null
251-
}
252-
"cmd" {
253-
if ($scenario.launch -eq "path") {
254-
Invoke-CliProcess -FilePath "cmd" -ArgumentList @("/d", "/s", "/c", "gh-aw --help") -EnvironmentOverrides $envOverrides -UnsetEnvironment $unsetVars -AssertNoAnsi:$assertNoAnsi | Out-Null
255-
} else {
256-
# Use RawArguments to bypass .NET's \"..\" quoting which conflicts with cmd.exe's ""..""
257-
# syntax. The outer "" pair is stripped by cmd /s, leaving "path\exe.exe" --help.
258-
Invoke-CliProcess -FilePath "cmd" -RawArguments "/d /s /c `"`"$binary`" --help`"" -EnvironmentOverrides $envOverrides -UnsetEnvironment $unsetVars -AssertNoAnsi:$assertNoAnsi | Out-Null
258+
switch ($scenario.shell) {
259+
"pwsh-default" {
260+
$cmd = if ($scenario.launch -eq "path") { "& gh-aw --help" } else { "& '$binary' --help" }
261+
Invoke-CliProcess -FilePath "pwsh" -ArgumentList @("-Command", $cmd) -EnvironmentOverrides $envOverrides -UnsetEnvironment $unsetVars -AssertNoAnsi:$assertNoAnsi | Out-Null
262+
}
263+
"pwsh-noprofile" {
264+
$cmd = if ($scenario.launch -eq "path") { "& gh-aw --help" } else { "& '$binary' --help" }
265+
Invoke-CliProcess -FilePath "pwsh" -ArgumentList @("-NoProfile", "-Command", $cmd) -EnvironmentOverrides $envOverrides -UnsetEnvironment $unsetVars -AssertNoAnsi:$assertNoAnsi | Out-Null
266+
}
267+
"powershell-default" {
268+
$cmd = if ($scenario.launch -eq "path") { "& gh-aw --help" } else { "& '$binary' --help" }
269+
Invoke-CliProcess -FilePath "powershell" -ArgumentList @("-Command", $cmd) -EnvironmentOverrides $envOverrides -UnsetEnvironment $unsetVars -AssertNoAnsi:$assertNoAnsi | Out-Null
270+
}
271+
"powershell-noprofile" {
272+
$cmd = if ($scenario.launch -eq "path") { "& gh-aw --help" } else { "& '$binary' --help" }
273+
Invoke-CliProcess -FilePath "powershell" -ArgumentList @("-NoProfile", "-Command", $cmd) -EnvironmentOverrides $envOverrides -UnsetEnvironment $unsetVars -AssertNoAnsi:$assertNoAnsi | Out-Null
274+
}
275+
"cmd" {
276+
if ($scenario.launch -eq "path") {
277+
Invoke-CliProcess -FilePath "cmd" -ArgumentList @("/d", "/s", "/c", "gh-aw --help") -EnvironmentOverrides $envOverrides -UnsetEnvironment $unsetVars -AssertNoAnsi:$assertNoAnsi | Out-Null
278+
} else {
279+
# Use RawArguments to bypass .NET's \"..\" quoting which conflicts with cmd.exe's ""..""
280+
# syntax. The outer "" pair is stripped by cmd /s, leaving "path\exe.exe" --help.
281+
Invoke-CliProcess -FilePath "cmd" -RawArguments "/d /s /c `"`"$binary`" --help`"" -EnvironmentOverrides $envOverrides -UnsetEnvironment $unsetVars -AssertNoAnsi:$assertNoAnsi | Out-Null
282+
}
283+
}
284+
default {
285+
throw "Unknown shell scenario: $($scenario.shell)"
259286
}
260287
}
261-
default {
262-
throw "Unknown shell scenario: $($scenario.shell)"
263-
}
288+
} catch {
289+
throw "Scenario '$($scenario.name)' failed: $($_.Exception.Message)"
264290
}
265291
266292
Add-Content $env:GITHUB_STEP_SUMMARY "✅ Matrix scenario passed: $($scenario.name)"
@@ -631,6 +657,9 @@ jobs:
631657
timeout-minutes: 5
632658
permissions:
633659
contents: read
660+
# Scoped to this job only (build/integration stay contents:read) and used
661+
# solely by the failure-issue step below to dedupe/create a tracking issue.
662+
issues: write
634663
steps:
635664
- name: Evaluate outcomes
636665
id: evaluate
@@ -643,6 +672,7 @@ jobs:
643672
if [[ "$INTEGRATION" != "success" ]]; then FAILED="${FAILED:+$FAILED, }integration($INTEGRATION)"; fi
644673
if [[ -n "$FAILED" ]]; then
645674
echo "failed=true" >> "$GITHUB_OUTPUT"
675+
echo "failed_jobs=$FAILED" >> "$GITHUB_OUTPUT"
646676
RUN_URL="${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
647677
{
648678
echo "❌ Failed jobs: $FAILED"
@@ -654,6 +684,40 @@ jobs:
654684
echo "✅ All Windows integration jobs passed" >> "$GITHUB_STEP_SUMMARY"
655685
fi
656686
687+
# Minimal-permission failure reporting: only this step (and only on failure)
688+
# uses issues:write. Reuses an existing open tracking issue when one exists
689+
# so repeated scheduled failures don't spam new issues (see #66441).
690+
- name: Create or update failure issue
691+
if: steps.evaluate.outputs.failed == 'true'
692+
env:
693+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
694+
FAILED_JOBS: ${{ steps.evaluate.outputs.failed_jobs }}
695+
run: |
696+
set -euo pipefail
697+
RUN_URL="${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
698+
TITLE="❌ Windows CLI integration failing"
699+
BODY=$(cat <<EOF
700+
## Windows CLI Integration Failure
701+
702+
**Failed jobs:** ${FAILED_JOBS}
703+
704+
**Run:** [${GITHUB_RUN_ID}](${RUN_URL})
705+
706+
**Date:** $(date -u '+%Y-%m-%d %H:%M UTC')
707+
708+
This issue was created automatically by the daily Windows CLI integration workflow and is reused for subsequent failures until closed.
709+
EOF
710+
)
711+
712+
EXISTING=$(gh issue list --repo "${GITHUB_REPOSITORY}" --state open --search "in:title \"${TITLE}\"" --json number --jq '.[0].number // empty')
713+
if [[ -n "$EXISTING" ]]; then
714+
gh issue comment "$EXISTING" --repo "${GITHUB_REPOSITORY}" --body "$BODY"
715+
echo "🔁 Updated existing tracking issue #${EXISTING}" >> "$GITHUB_STEP_SUMMARY"
716+
else
717+
NEW_URL=$(gh issue create --repo "${GITHUB_REPOSITORY}" --title "$TITLE" --body "$BODY" --label "bug")
718+
echo "🆕 Created tracking issue: ${NEW_URL}" >> "$GITHUB_STEP_SUMMARY"
719+
fi
720+
657721
- name: Fail if any job failed
658722
if: steps.evaluate.outputs.failed == 'true'
659723
run: exit 1

0 commit comments

Comments
 (0)