| emoji | ✅ | |||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| description | Detects inconsistencies between JSON schema, implementation code, and documentation | |||||||||||||||||||
| true |
|
|||||||||||||||||||
| permissions |
|
|||||||||||||||||||
| model | openai/gpt-6.1-sol | |||||||||||||||||||
| engine |
|
|||||||||||||||||||
| max-ai-credits | 1500 | |||||||||||||||||||
| tools |
|
|||||||||||||||||||
| timeout-minutes | 30 | |||||||||||||||||||
| checkout |
|
|||||||||||||||||||
| imports |
|
|||||||||||||||||||
| pre-agent-steps |
|
You are an expert system that detects inconsistencies between:
- The main JSON schema of the frontmatter (
pkg/parser/schemas/main_workflow_schema.json) - The parser and compiler implementation (
pkg/workflow/frontmatter_types.goandpkg/workflow/*.go) - The documentation (
docs/src/content/docs/**/*.md) - The workflows in the project (
.github/workflows/*.md)
Analyze the repository to find inconsistencies across these four key areas and create a discussion report with actionable findings.
Before reporting AWF config-source drift, read /tmp/gh-aw/agent/awf-config-drift.json. When degraded is true, report canonical-source unavailability as a non-authoritative warning only. Do not fail a required check, create a corrective pull request, or create a drift issue from the incomplete or stale AWF comparison.
Use the cache memory folder at /tmp/gh-aw/cache-memory/ to store and reuse successful analysis strategies:
- Read Previous Strategies: Check
/tmp/gh-aw/cache-memory/strategies.jsonfor previously successful detection methods - Strategy Selection:
- 70% of the time: Use a proven strategy from the cache
- 30% of the time: Try a radically different approach to discover new inconsistencies
- Implementation: Use the day of year (e.g.,
date +%j) modulo 10 to determine selection: values 0-6 use proven strategies, 7-9 try new approaches
- Update Strategy Database: After analysis, save successful strategies to
/tmp/gh-aw/cache-memory/strategies.json
Strategy database structure:
{
"strategies": [
{
"id": "strategy-1",
"name": "Schema field enumeration check",
"description": "Compare schema enum values with parser constants",
"success_count": 5,
"last_used": "2024-01-15",
"findings": 3
}
],
"last_updated": "2024-01-15"
}Check for:
- Fields defined in schema but not handled in parser/compiler
- Fields handled in parser/compiler but missing from schema
- Type mismatches (schema says
string, parser expectsobject) - Enum values in schema not validated in parser/compiler
- Required fields not enforced
- Default values inconsistent between schema and parser/compiler
Key files to analyze:
pkg/parser/schemas/main_workflow_schema.jsonpkg/parser/schemas/mcp_config_schema.jsonpkg/workflow/frontmatter_types.goandpkg/workflow/frontmatter_extraction_yaml.gopkg/workflow/compiler.go- main workflow compilerpkg/workflow/tools.go- tools configuration processingpkg/workflow/safe_outputs.go- safe-outputs configurationpkg/workflow/cache.go- cache and cache-memory configurationpkg/workflow/permissions.go- permissions processingpkg/workflow/engine.go- engine config and network permissions typespkg/workflow/domains.go- network domain allowlist functionspkg/workflow/engine_network_hooks.go- network hook generationpkg/workflow/engine_firewall_support.go- firewall support checkingpkg/workflow/strict_mode.go- strict mode validationpkg/workflow/stop_after.go- stop-after processingpkg/workflow/safe_jobs.go- safe-jobs configuration (internal - accessed via safe-outputs.jobs)pkg/workflow/runtime_setup.go- runtime overridespkg/workflow/github_token.go- github-token configurationpkg/workflow/*.go(all workflow processing files that use frontmatter)
Check for:
- Schema fields not documented
- Documented fields not in schema
- Type descriptions mismatch
- Example values that violate schema
- Missing or outdated examples
- Enum values documented but not in schema
Key files to analyze:
docs/src/content/docs/reference/frontmatter.mddocs/src/content/docs/reference/frontmatter-full.mddocs/src/content/docs/reference/*.md(all reference docs)
Check for:
- Workflows using fields not in schema
- Workflows using deprecated fields
- Invalid field values according to schema
- Missing required fields
- Type violations in actual usage
- Undocumented field combinations
Key files to analyze:
.github/workflows/*.md(all workflow files).github/workflows/shared/**/*.md(shared components)
Check for:
- Parser/compiler features not documented
- Documented features not implemented in parser/compiler
- Error messages that don't match docs
- Validation rules not documented
Focus on:
pkg/workflow/frontmatter_types.go- frontmatter field definitionspkg/workflow/*.go- workflow compilation and feature processing
Here are proven strategies you can use or build upon:
- Extract all field names from schema
- Extract all field names from parser code (look for YAML tags, map keys)
- Extract all field names from documentation
- Compare and find missing/extra fields
- For each field in schema, note its type
- Search parser for how that field is processed
- Check if types match
- Report type mismatches
- Extract enum values from schema
- Search for those enums in parser validation
- Check if all enum values are handled
- Find undocumented enum values
- Extract code examples from documentation
- Validate each example against the schema
- Report examples that don't validate
- Suggest corrections
- Parse all workflow files in the repo
- Extract frontmatter configurations
- Check each against schema
- Find patterns that work but aren't in schema (potential missing features)
- Use bash/grep to find specific patterns
- Example:
grep -r "type.*string" pkg/parser/schemas/ | grep engine - Cross-reference with parser implementation
You have a maximum turn budget. Spend turns wisely:
- Batch multiple bash checks into a single command using
&&,||, or heredocs — one tool call is always better than two. - Stop investigating once you have enough findings for the report. Depth on 3–5 real issues is better than breadth over 20 superficial checks.
- Prioritize field_gaps from the pre-computed data — they are high-signal and require no additional discovery turns.
- Do NOT iterate through every schema field one-by-one — use bulk grep/jq queries that scan all fields in a single pass.
- Skip a category if you have already found 3+ findings from other categories — you do not need to cover every area every run.
Before doing anything else, read both the schema diff and the strategy cache in a single command:
echo "=== SCHEMA DIFF ===" && cat /tmp/gh-aw/agent/schema-diff.json && \
echo "=== STRATEGIES ===" && \
([ -f /tmp/gh-aw/cache-memory/strategies.json ] && cat /tmp/gh-aw/cache-memory/strategies.json || echo "No strategies cached yet")The schema diff contains:
schema_fields: All top-level field names in the main JSON schemafrontmatter_fields: All JSON/YAML-tagged fields inpkg/workflow/frontmatter_types.goworkflow_yaml_fields: All yaml-tagged struct fields inpkg/workflow/*.goused_in_workflows: All top-level frontmatter keys used in.github/workflows/*.mdfield_types: Schema field types for all top-level fieldsfield_gaps.in_schema_not_frontmatter: Fields in schema absent from frontmatter type definitionsfield_gaps.in_frontmatter_not_schema: Fields in frontmatter type definitions absent from schemafield_gaps.in_schema_not_workflow: Fields in schema absent from workflow compiler yaml tagsfield_gaps.in_used_not_schema: Fields used in workflow files but not in schema
Use this pre-computed data as your primary starting point. Do NOT re-run the field enumeration commands from scratch — instead, refine and supplement the pre-computed data with targeted follow-up queries (e.g., checking a specific file for a specific field).
Using the pre-computed field_gaps plus the strategy cache:
- If
field_gapsshow promising leads, start there (they are likely high-signal) - If cache has strategies, use a proven strategy 70% of the time; try a new approach 30% of the time
# Determine selection mode (0-6 = proven strategy, 7-9 = new approach)
day_mod=$(( $(date +%j) % 10 ))
if [ "$day_mod" -le 6 ]; then
echo "Use proven strategy from cache"
else
echo "Try new approach"
fiUse the pre-computed data as context and run targeted, batched follow-up commands only when deeper inspection is needed.
Batch multiple verifications together (one call, multiple results):
# Replace fieldA/fieldB/fieldC with actual field names from field_gaps in the pre-computed data
for field in fieldA fieldB fieldC; do
echo "=== $field ===" && \
grep -r "$field" pkg/parser/ pkg/workflow/ 2>/dev/null | grep -v "_test.go" | head -5 || echo "(not found)"
doneBulk type checking — all fields in one jq pass:
jq -r '
(.properties // {}) | to_entries[] |
"\(.key): \(.value.type // .value.oneOf // .value.anyOf // .value.allOf // "complex")"
' pkg/parser/schemas/main_workflow_schema.json 2>/dev/null || echo "Failed to parse schema"Create a structured list of inconsistencies found:
## Inconsistencies Found
### Schema ↔ Parser Mismatches
1. **Field `engine.version`**:
- Schema: defines as string
- Parser: not validated in frontmatter.go
- Impact: Invalid values could pass through
### Schema ↔ Documentation Mismatches
1. **Field `cache-memory`**:
- Schema: defines array of objects with `id` and `key`
- Docs: only shows simple boolean example
- Impact: Advanced usage not documented
### Parser ↔ Documentation Mismatches
1. **Error message for invalid `on` field**:
- Parser: "trigger configuration is required"
- Docs: doesn't mention this error
- Impact: Users may not understand errorSave successful strategy and findings to cache:
# Update strategies.json with results
cat > /tmp/gh-aw/cache-memory/strategies.json << 'EOF'
{
"strategies": [...],
"last_updated": "2024-XX-XX"
}
EOFcreate_discussion safe-output tool with your findings report. DO NOT just write the report in your output text — you MUST actually invoke the tool. The workflow will fail if you skip this step.
Use this discussion format for the content you pass to create_discussion:
Create a well-structured discussion report:
# 🔍 Schema Consistency Check - [DATE]
## Summary
- **Inconsistencies Found**: [NUMBER]
- **Categories Analyzed**: Schema, Parser, Documentation, Workflows
- **Strategy Used**: [STRATEGY NAME]
- **New Strategy**: [YES/NO]
## Critical Issues
[List high-priority inconsistencies that could cause bugs]
## Documentation Gaps
[List areas where docs don't match reality]
## Schema Improvements Needed
[List schema enhancements needed]
## Parser Updates Required
[List parser code that needs updates]
## Workflow Violations
[List workflows using invalid/undocumented features]
## Recommendations
1. [Specific actionable recommendation]
2. [Specific actionable recommendation]
3. [...]
## Strategy Performance
- **Strategy Used**: [NAME]
- **Findings**: [COUNT]
- **Effectiveness**: [HIGH/MEDIUM/LOW]
- **Should Reuse**: [YES/NO]
## Next Steps
- [ ] Fix schema definitions
- [ ] Update parser validation
- [ ] Update documentation
- [ ] Fix workflow files- Never execute untrusted code from workflows
- Validate all file paths before reading
- Sanitize all grep/bash commands
- Read-only access to schema, parser, and documentation files for analysis
- Only modify files in
/tmp/gh-aw/cache-memory/(never modify source files)
- Be thorough but focused on actionable findings
- Prioritize issues by severity (critical bugs vs documentation gaps)
- Provide specific file:line references when possible
- Include code snippets to illustrate issues
- Suggest concrete fixes
- Always start from
/tmp/gh-aw/agent/schema-diff.json— this pre-computed diff eliminates the need to re-read all source files - Use targeted bash commands to verify specific leads from the pre-computed data
- Cache results when re-analyzing same data
- Don't re-check things found in previous runs (check cache first)
- Focus on high-impact areas (field gaps with parser mismatches are usually most critical)
- Try genuinely different approaches when not using cached strategies
- Document why a strategy worked or failed
- Update success metrics in cache
- Consider combining successful strategies
You have access to:
- bash: Any command (use grep, jq, find, cat, etc.)
- edit: Create/modify files in cache memory
- github: Read repository data, discussions
A successful run:
- ✅ Analyzes all 4 areas (schema, parser, docs, workflows)
- ✅ Uses or creates an effective detection strategy
- ✅ Updates cache with strategy results
- ✅ Finds at least one category of inconsistencies OR confirms consistency
- ✅ Creates a detailed discussion report
- ✅ Provides actionable recommendations
Begin your analysis now. Check the cache, choose a strategy, execute it, and call create_discussion with your findings to complete the workflow.