Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/threat-detection-report-as-issue-option.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

17 changes: 17 additions & 0 deletions docs/src/content/docs/reference/threat-detection.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,8 @@ safe-outputs:
| `steps` | array | Additional GitHub Actions steps to run **before** AI analysis (pre-steps) |
| `post-steps` | array | Additional GitHub Actions steps to run **after** AI analysis (post-steps) |
| `max-ai-credits` | integer | AI Credits cap for the detection run, independent of the main agent budget. Defaults to `400` when unset, with runtime override via `vars.GH_AW_DEFAULT_DETECTION_MAX_AI_CREDITS`. Accepts plain integers; `-1` disables the detection budget. |
| `continue-on-error` | boolean | When `true` (default), detection warnings/failures produce a caution notice instead of blocking safe outputs. |
| `report-as-issue` | boolean | When `true` (default), detection warnings/failures create or update the `[aw] Detection Runs` tracking issue. Set to `false` to keep threat detection and its enforcement enabled while skipping the tracking issue; results remain visible in the GitHub Actions run logs. |

## Detection Budget

Expand Down Expand Up @@ -378,6 +380,21 @@ If the detection process itself fails (e.g., network issues, tool errors), the w

A warning is a lower-severity signal than a hard threat: the safe output is allowed to proceed, but human review is required before merge. When `create-pull-request` is the safe output, the handler submits a `REQUEST_CHANGES` pull request review whose body includes the detection reason and a link to the workflow run logs. If a `request_review` protected-files gate also fires in the same run, both signals are composed into a single review body separated by a horizontal rule.

**Opting Out of Tracking Issues:**

By default, a warning or failure conclusion also creates or updates a `[aw] Detection Runs` tracking issue in the repository and posts a comment describing the run. This is useful for auditing automation health, but not every repository wants detector diagnostics (such as `parse_error` reports) surfaced alongside user-facing issues.

Set `report-as-issue: false` to keep threat detection and its enforcement fully active while skipping the tracking issue entirely:

```yaml wrap
safe-outputs:
create-issue:
threat-detection:
report-as-issue: false
```

Detection still runs, `continue-on-error` behavior is unchanged, and results remain available in the GitHub Actions run summary and logs.
Comment thread
Copilot marked this conversation as resolved.
Resolved

## Supply Chain Protection (Protected Files)

Beyond AI-powered threat detection, GitHub Agentic Workflows includes a static, rule-based protection layer that guards against **supply chain attacks** — cases where an AI agent could (intentionally or accidentally) modify files that control how software is built, tested, or deployed.
Expand Down
5 changes: 5 additions & 0 deletions pkg/parser/schemas/main_workflow_schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -11911,6 +11911,11 @@
"continue-on-error": {
"$ref": "#/$defs/templatable_boolean",
"description": "When true (default), detection failures produce warnings and allow safe outputs to proceed with a caution notice and 'needs-review' label. When false, detection failures block safe outputs entirely. Accepts a boolean literal or a GitHub Actions expression."
},
"report-as-issue": {
"type": "boolean",
"default": true,
"description": "When true (default), detection warnings/failures create or update the '[aw] Detection Runs' tracking issue and post a comment to it. When false, detection still runs and enforces its configured continue-on-error behavior, but no tracking issue is created or updated; results remain available in the GitHub Actions run diagnostics."
Comment on lines +11915 to +11918

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regenerated frontmatter-full.md from the schema, including the new report-as-issue option. Addressed in 03afa9f.

}
},
"additionalProperties": false
Expand Down
59 changes: 59 additions & 0 deletions pkg/workflow/detection_success_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import (
"strings"
"testing"

"github.com/github/gh-aw/pkg/constants"
"github.com/github/gh-aw/pkg/stringutil"

"github.com/github/gh-aw/pkg/testutil"
Expand Down Expand Up @@ -236,6 +237,64 @@ Create an issue.
}
}

func TestDetectionRunsStepOmittedWhenReportAsIssueDisabled(t *testing.T) {
tmpDir := testutil.TempDir(t, "test-*")
workflowPath := filepath.Join(tmpDir, "test-workflow.md")

frontmatter := `---
on: workflow_dispatch
permissions:
contents: read
engine: claude
safe-outputs:
create-issue:
threat-detection:
report-as-issue: false
---

# Test

Create an issue.
`

if err := os.WriteFile(workflowPath, []byte(frontmatter), 0644); err != nil {
t.Fatalf("Failed to write workflow file: %v", err)
}

compiler := NewCompiler()
if err := compiler.CompileWorkflow(workflowPath); err != nil {
t.Fatalf("Failed to compile: %v", err)
}

// Read the compiled YAML
lockPath := stringutil.MarkdownToLockFile(workflowPath)
yamlBytes, err := os.ReadFile(lockPath)
if err != nil {
t.Fatalf("Failed to read compiled YAML: %v", err)
}
yaml := string(yamlBytes)

// Verify conclusion job exists (other conclusion steps still run)
conclusionSection := extractJobSection(yaml, "conclusion")
if conclusionSection == "" {
t.Fatal("Conclusion job not found in compiled YAML")
}

// The detection runs tracking issue step must be omitted entirely.
if strings.Contains(conclusionSection, "Log detection run") {
t.Error("Conclusion job should not contain 'Log detection run' step when report-as-issue is disabled")
}
if strings.Contains(conclusionSection, "handle_detection_runs.cjs") {
t.Error("Conclusion job should not reference handle_detection_runs.cjs when report-as-issue is disabled")
}

// Detection job itself must still be present since threat detection remains enabled.
detectionSection := extractJobSection(yaml, string(constants.DetectionJobName))
if detectionSection == "" {
t.Error("Detection job should still be compiled when only report-as-issue is disabled")
}
}

func TestNoopStepGetsThreatDetectionAICEnvVar(t *testing.T) {
tmpDir := testutil.TempDir(t, "test-*")
workflowPath := filepath.Join(tmpDir, "test-workflow.md")
Expand Down
6 changes: 6 additions & 0 deletions pkg/workflow/notify_comment_conclusion_helpers.go
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,12 @@ func (c *Compiler) buildConclusionDetectionRunsStep(data *WorkflowData, mainJobN
if !IsDetectionJobEnabled(data.SafeOutputs) {
return nil
}
// Allow opting out of the "[aw] Detection Runs" tracking issue independently of
// threat detection itself via safe-outputs.threat-detection.report-as-issue: false.
if data.SafeOutputs != nil && data.SafeOutputs.ThreatDetection != nil && !data.SafeOutputs.ThreatDetection.IsReportAsIssueEnabled() {
notifyCommentLog.Print("Skipping detection runs logging step: report-as-issue is disabled")
return nil
}
envVars := buildWorkflowMetadataEnvVarsWithTrackerID(data.Name, data.Source, data.TrackerID, buildLocalWorkflowSourceURL(c.markdownPath))
envVars = append(envVars, " GH_AW_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\n")
envVars = append(envVars, fmt.Sprintf(" GH_AW_DETECTION_CONCLUSION: ${{ needs.%s.outputs.detection_conclusion }}\n", constants.DetectionJobName))
Expand Down
80 changes: 58 additions & 22 deletions pkg/workflow/threat_detection_config.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ type ThreatDetectionConfig struct {
RunsOn string `yaml:"runs-on,omitempty"` // Runner override for the detection job
Environment string `yaml:"environment,omitempty"` // GitHub Actions environment override for the detection job (defaults to top-level environment when OIDC is used)
ContinueOnError *bool `yaml:"continue-on-error,omitempty"` // When true (default), detection failures produce warnings instead of blocking safe outputs
ReportAsIssue *bool `yaml:"report-as-issue,omitempty"` // When true (default), detection warnings/failures create/update the "[aw] Detection Runs" tracking issue
EnabledExpr *string `yaml:"-"` // Expression form of the enabled flag, e.g. "${{ inputs.enable-threat-detection }}"
ContinueOnErrorExpr *string `yaml:"-"` // Expression form of continue-on-error, e.g. "${{ inputs.coe }}"
}
Expand All @@ -33,6 +34,13 @@ func (td *ThreatDetectionConfig) IsContinueOnError() bool {
return td.ContinueOnError == nil || *td.ContinueOnError
}

// IsReportAsIssueEnabled reports whether detection warnings/failures should create
// or update the "[aw] Detection Runs" tracking issue. Defaults to true (enabled)
// when not explicitly set, preserving existing behaviour.
func (td *ThreatDetectionConfig) IsReportAsIssueEnabled() bool {
return td == nil || td.ReportAsIssue == nil || *td.ReportAsIssue
}

// HasRunnableDetection reports whether this config will produce a detection job
// that actually executes. Returns false when the engine is disabled and no
// custom steps are configured, since the job would have nothing to run.
Expand Down Expand Up @@ -118,6 +126,17 @@ func (c *Compiler) parseThreatDetectionConfig(outputMap map[string]any) *ThreatD
func (c *Compiler) parseThreatDetectionObjectConfig(configMap map[string]any) *ThreatDetectionConfig {
threatConfig := &ThreatDetectionConfig{}

parseThreatDetectionScalarFields(configMap, threatConfig)
c.parseThreatDetectionEngineField(configMap, threatConfig)

threatLog.Printf("Threat detection configured with custom prompt: %v, custom pre-steps: %v, custom post-steps: %v", threatConfig.Prompt != "", len(threatConfig.Steps) > 0, len(threatConfig.PostSteps) > 0)
return threatConfig
}

// parseThreatDetectionScalarFields parses the non-engine fields of the threat-detection
// object configuration (prompt, steps, post-steps, budgets, runner, and reporting flags)
// into threatConfig.
func parseThreatDetectionScalarFields(configMap map[string]any, threatConfig *ThreatDetectionConfig) {
// Parse prompt field
if prompt, exists := configMap["prompt"]; exists {
if promptStr, ok := prompt.(string); ok {
Expand Down Expand Up @@ -170,6 +189,13 @@ func (c *Compiler) parseThreatDetectionObjectConfig(configMap map[string]any) *T
threatConfig.RunsOn = renderRunsOnSnippet(runOn)
}

parseThreatDetectionReportingFields(configMap, threatConfig)
}

// parseThreatDetectionReportingFields parses the fields that control how detection
// results are enforced and reported (continue-on-error and report-as-issue) into
// threatConfig.
func parseThreatDetectionReportingFields(configMap map[string]any, threatConfig *ThreatDetectionConfig) {
// Parse continue-on-error field (default: true).
// Accepts a literal bool or a GitHub Actions expression string.
if coe, exists := configMap["continue-on-error"]; exists {
Expand All @@ -185,31 +211,41 @@ func (c *Compiler) parseThreatDetectionObjectConfig(configMap map[string]any) *T
}
}

// Parse engine field (supports string, object, and boolean false formats)
if engine, exists := configMap["engine"]; exists {
// Handle boolean false to disable AI engine
if engineBool, ok := engine.(bool); ok {
if !engineBool {
threatLog.Print("Threat detection AI engine disabled")
// engine: false means no AI engine steps
threatConfig.EngineConfig = nil
threatConfig.EngineDisabled = true
}
} else if engineStr, ok := engine.(string); ok {
threatLog.Printf("Threat detection engine set to: %s", engineStr)
// Handle string format
threatConfig.EngineConfig = &EngineConfig{ID: engineStr}
} else if engineObj, ok := engine.(map[string]any); ok {
threatLog.Print("Parsing threat detection engine configuration")
// Handle object format - use extractEngineConfig logic
_, engineConfig, model := c.ExtractEngineConfig(map[string]any{"engine": engineObj})
threatConfig.EngineConfig = engineConfig
threatConfig.Model = model
// Parse report-as-issue field (default: true).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] parseThreatDetectionObjectConfig's table-driven test (TestParseThreatDetectionConfig) covers every other reporting/kill-switch field but has no case for report-as-issue, so parsing regressions here (e.g. accidentally dropping the field, or breaking it when combined with continue-on-error) wouldn't be caught at the parser-unit level — only by the heavier compile-level test in detection_success_test.go.

💡 Suggested addition to TestParseThreatDetectionConfig
{
	name: "object with report-as-issue false",
	outputMap: map[string]any{
		"threat-detection": map[string]any{
			"report-as-issue": false,
		},
	},
	expectedConfig: &ThreatDetectionConfig{
		ReportAsIssue: boolPtr(false),
	},
},

This keeps parser-level coverage symmetric with continue-on-error and catches regressions closer to the source than the full-compile test.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added parser-level coverage for report-as-issue: false combined with continue-on-error: false. Addressed in 03afa9f.

if reportAsIssue, exists := configMap["report-as-issue"]; exists {
if boolVal, ok := reportAsIssue.(bool); ok {
threatConfig.ReportAsIssue = &boolVal
threatLog.Printf("Threat detection report-as-issue set to: %v", boolVal)
}
}
}

threatLog.Printf("Threat detection configured with custom prompt: %v, custom pre-steps: %v, custom post-steps: %v", threatConfig.Prompt != "", len(threatConfig.Steps) > 0, len(threatConfig.PostSteps) > 0)
return threatConfig
// parseThreatDetectionEngineField parses the engine field (supports string, object, and
// boolean false formats) into threatConfig.
func (c *Compiler) parseThreatDetectionEngineField(configMap map[string]any, threatConfig *ThreatDetectionConfig) {
engine, exists := configMap["engine"]
if !exists {
return
}
// Handle boolean false to disable AI engine
if engineBool, ok := engine.(bool); ok {
if !engineBool {
threatLog.Print("Threat detection AI engine disabled")
// engine: false means no AI engine steps
threatConfig.EngineConfig = nil
threatConfig.EngineDisabled = true
}
} else if engineStr, ok := engine.(string); ok {
threatLog.Printf("Threat detection engine set to: %s", engineStr)
// Handle string format
threatConfig.EngineConfig = &EngineConfig{ID: engineStr}
} else if engineObj, ok := engine.(map[string]any); ok {
threatLog.Print("Parsing threat detection engine configuration")
// Handle object format - use extractEngineConfig logic
_, engineConfig, model := c.ExtractEngineConfig(map[string]any{"engine": engineObj})
threatConfig.EngineConfig = engineConfig
threatConfig.Model = model
}
}

// extractRawExpression strips the "${{" prefix and "}}" suffix from a GitHub Actions
Expand Down
Loading