Skip to content

Fix Claude persona evaluator registration #66634

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pelikhan merged 1 commit into from
Oct 7, 2026
Merged

Fix Claude persona evaluator registration #66634

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 19 additions & 12 deletions .github/aw/subagents.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,26 +31,33 @@ and key contents. Be concise and factual.

### Block boundary

The block ends at the next `##` heading (any level-2 heading) or at EOF — no explicit end marker is needed. Place sub-agent blocks **at the bottom** of the file, after all main workflow content.
The block ends at the next `##` heading (any level-2 heading) or at EOF. Place sub-agent blocks **at the bottom** of the file, after all main workflow content. If parent instructions follow a sub-agent, close it with ``## end agent: `name` `` so those instructions remain in the main prompt.

@github-actions github-actions Bot Oct 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/diagnosing-bugs] Nested backtick code span (## end agent: `name` ) will render incorrectly in GitHub-flavored Markdown — GFM closes the span at the first matching double-backtick sequence, so this is likely to display broken.

💡 Suggested fix

Use a single backtick span with the literal name escaped, or switch to a fenced inline example:

close it with `## end agent: \`name\`` so those instructions remain in the main prompt.

or simply drop the inner backticks around name since it is already inside the outer span.

@copilot please address this.


### Frontmatter fields

Only two fields are supported inside a sub-agent frontmatter block:
Sub-agent frontmatter is preserved as authored. Use fields and model values supported by the workflow's engine; gh-aw does not translate model aliases or insert the heading's name into the frontmatter.

| Field | Required | Default | Notes |
|---|---|---|---|
| `description` | No | — | Human-readable summary of the sub-agent's role |
| `model` | No | `"inherited"` | Model override; `"inherited"` uses the parent workflow's model. Prefer model aliases (e.g. `small`, `large`) over specific model IDs for portability. |
| `name` | Claude: yes | — | Claude's registered agent identifier. Use the same name as the inline heading. |

@github-actions github-actions Bot Oct 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/grill-with-docs] Good catch documenting Claude's silent-skip behavior and the name requirement. However, the unchanged examples later in this same file (dependency-scanner, test-coverage, secret-scanner, diff-explainer under "When to Use Sub-Agents" / "Full Example") still omit name: in their frontmatter, now contradicting the rule just introduced here.

💡 Suggested fix

Either (a) add name: to all example blocks for consistency, or (b) add a one-line caveat noting those examples target the Copilot engine (which does not require name) so readers don't copy a pattern that silently breaks on Claude.

@copilot please address this.

| `description` | Claude: yes | — | Human-readable summary of the sub-agent's role |
| `model` | No | Engine default | Engine-native model override. For Claude, `inherit` uses the parent model; `inherited` is not the inheritance sentinel. |

Built-in aliases resolve to the best available model per provider and keep working as models are updated. Common sub-agent aliases:
For Claude, include both required fields and use its native inheritance value:

| Alias | Resolves to | When to use |
|---|---|---|
| `small` | `mini` → haiku, gpt-5-mini, gpt-5-nano, gemini-flash | Cheap, fast tasks: extraction, classification, formatting |
| `large` | sonnet, gpt-5-pro, gpt-5, gemini-pro | Complex reasoning or synthesis tasks |
| `inherited` | Parent workflow model | Default — use when the sub-agent needs the same capability as the parent |
```markdown
## agent: `file-summarizer`
---
name: file-summarizer
description: Summarizes a file
model: inherit
---
Read the requested file and return a concise summary.
```

Claude silently skips project agent files without `name`, even when the filename matches the inline heading. See [Claude's sub-agent frontmatter reference](https://code.claude.com/docs/en/sub-agents#supported-frontmatter-fields).

All other fields (`engine`, `tools`, `network`, etc.) are stripped at runtime with a warning. Sub-agents inherit the parent's engine, tool access, and network configuration.
Aliases such as `small` in the Copilot examples below are engine-specific, not portable substitutions. Other frontmatter fields are also preserved; native tool restrictions must use the engine's syntax. Sub-agents run within the parent's engine and network configuration.

---

Expand Down Expand Up @@ -199,6 +206,6 @@ changes. Return a bulleted list, one bullet per file.

## Limitations

- Sub-agents do not support `engine:`, `tools:`, `network:`, or `mcp-servers:` fields — those are stripped at runtime.
- gh-aw does not interpret sub-agent `engine:`, `tools:`, `network:`, or `mcp-servers:` fields as workflow configuration. Preserved fields are interpreted by the engine, if supported.
- Sub-agents cannot define their own safe-output jobs.
- Sub-agent blocks must appear in the main workflow file body; they are not resolved inside imported shared files.
4 changes: 2 additions & 2 deletions .github/workflows/agent-persona-explorer.lock.yml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 4 additions & 1 deletion .github/workflows/agent-persona-explorer.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -323,13 +323,16 @@ Store the structured records in `/tmp/gh-aw/cache-memory/agent-persona-explorer/

## agent: `persona-evaluator`
---
name: persona-evaluator
description: Evaluates workflow scenarios using the repository's agentic-workflows authoring guidance
model: inherited
model: inherit
---
You are acting as the `agentic-workflows` custom agent for this persona evaluation. Read `.github/agents/agentic-workflows.md` and follow the local ad hoc-evaluation guidance in `.github/aw/create-agentic-workflow.md` for each scenario. Do not create or edit workflow files.

Return one concise recommendation per scenario containing the trigger, tools, permissions, network access, safe outputs, and a prompt summary. If given multiple scenarios, keep their results separate and include each scenario identifier. Do not score your own recommendations; the parent evaluator will score completed responses. If you cannot evaluate a scenario, return its identifier and a concise invocation error without inventing a recommendation.

## end agent: `persona-evaluator`

Comment on lines +334 to +335
## Success Criteria

Your effectiveness is measured by:
Expand Down
40 changes: 40 additions & 0 deletions pkg/cli/agent_persona_explorer_workflow_contract_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
//go:build !integration

package cli

import (
"os"
"path/filepath"
"testing"

"github.com/github/gh-aw/pkg/gitutil"
"github.com/github/gh-aw/pkg/parser"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)

func TestAgentPersonaExplorerWorkflowSubAgentContract(t *testing.T) {
t.Parallel()
repoRoot, err := gitutil.FindGitRoot()
if err != nil {
t.Skipf("Skipping test: not in a git repository: %v", err)
}

content, err := os.ReadFile(filepath.Join(repoRoot, ".github", "workflows", "agent-persona-explorer.md"))
require.NoError(t, err)

@github-actions github-actions Bot Oct 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new regression test only exercises the Go parser, so it would still pass if the JavaScript runtime extractor regressed again and wrote a Claude agent without the required name/model fields.

💡 Cover the runtime write path that actually failed

The production failure happened in the interpolate_prompt/extract_inline_sub_agents.cjs path that writes the engine-specific agent file, but this test stops at parser.ExtractInlineSubAgents and ExtractFrontmatterFromContent. That gives you a green test even if the JS extractor starts stripping or rewriting the frontmatter again.

A tighter regression would add a case next to actions/setup/js/extract_inline_sub_agents.test.cjs that runs writeInlineSubAgents(...) for the persona workflow and asserts the generated Claude file still contains name: persona-evaluator and model: inherit, and still excludes ## Success Criteria.


main, agents, err := parser.ExtractInlineSubAgents(string(content))
require.NoError(t, err)
require.Len(t, agents, 1)
require.Equal(t, "persona-evaluator", agents[0].Name)
assert.Contains(t, main, "## Success Criteria", "Extraction must preserve the parent's publishing instructions")

agent, err := parser.ExtractFrontmatterFromContent(agents[0].Content)
require.NoError(t, err)
assert.Equal(t, agents[0].Name, agent.Frontmatter["name"], "Claude requires a name in the generated agent's frontmatter")
assert.NotEmpty(t, agent.Frontmatter["description"], "Claude requires a description to register the agent")
assert.Equal(t, "inherit", agent.Frontmatter["model"], "Claude uses inherit, not inherited, for the parent model")
assert.Contains(t, agent.Markdown, ".github/agents/agentic-workflows.md")
assert.Contains(t, agent.Markdown, ".github/aw/create-agentic-workflow.md")
assert.NotContains(t, agent.Markdown, "## Success Criteria", "Parent instructions must stay outside the evaluator")
}
Loading