Skip to content

Commit 1c2c23b

Browse files
Copilotpelikhan
Copilot
and
pelikhan
authored
Refactor Tavily MCP configuration to shared workflow and add mcp-servers import support (#1280)
* Initial plan * Refactor Tavily MCP configuration to shared workflow and fix import merging - Created shared/tavily-mcp.md for reusable Tavily MCP configuration - Updated scout.md to use imports field instead of inline mcp-servers - Fixed bug in mergeTools where newline-separated JSON objects weren't being parsed correctly - Import functionality now properly handles multiple JSON objects from different imports Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * Add integration tests for imports functionality Added comprehensive tests to verify that imports work correctly with single and multiple imported files. Tests validate that MCP configurations are properly merged from imported files. Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * Complete: Tavily MCP refactoring with imports and tests Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * Add support for importing mcp-servers from shared workflows - Updated shared/tavily-mcp.md to use mcp-servers instead of tools - Added mcp-servers to included_file_schema.json - Added MergedMCPServers field to ImportsResult struct - Implemented extractMCPServersFromContent and mergeMCPServers functions - Updated ProcessImportsFromFrontmatterWithManifest to extract and merge mcp-servers - Updated compiler to merge imported mcp-servers with top-level mcp-servers - Added test for importing mcp-servers (TestCompileWorkflowWithMCPServersImport) Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * Address PR feedback: improve MCP servers schema and documentation - Copied full MCP servers schema from main workflow to included file schema with $defs for stdio_mcp_tool and http_mcp_tool - Refactored mergeMCPServers to ensure clean map creation instead of modifying input - Updated documentation to mention MCP servers can be imported: - include-directives.md: Added frontmatter merging section for mcp-servers with example - packaging-imports.md: Added MCP server import examples and updated existing examples - All tests passing Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * Merge main and recompile all workflows - Merged latest changes from main branch - Recompiled all workflows to update lock files - All tests passing including import-related tests - Tavily MCP server still correctly configured in scout workflow Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * Refactor discussion and issue creation scripts for improved readability and error handling - Simplified the parsing and validation of agent output in create_discussion.js and create_issue.js. - Enhanced error messages for better debugging. - Improved the handling of discussion categories and issue labels. - Added checks for empty titles and bodies, ensuring defaults are set appropriately. - Streamlined the creation process for discussions and issues, including better logging of created items. - Updated summary writing to include details of created discussions and issues. * Refactor: Move import/merge code to imports.go and fix linting - Created pkg/workflow/imports.go with mergeTools and mergeMCPServers functions - Removed these functions from compiler.go for better code organization - Fixed linting issue: removed unnecessary nil check in mergeMCPServers (S1031) - All tests passing including import-related tests - Scout workflow still compiles correctly with Tavily MCP server Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> Co-authored-by: Peli de Halleux <pelikhan@users.noreply.github.com>
1 parent ecc8a96 commit 1c2c23b

10 files changed

Lines changed: 652 additions & 37 deletions

File tree

‎.github/workflows/scout.lock.yml‎

Lines changed: 7 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎.github/workflows/scout.md‎

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -13,13 +13,11 @@ permissions:
1313
actions: read
1414
roles: [admin, maintainer, write]
1515
engine: copilot
16+
imports:
17+
- shared/tavily-mcp.md
1618
tools:
1719
cache-memory:
1820
retention-days: 7
19-
mcp-servers:
20-
tavily:
21-
url: "https://mcp.tavily.com/mcp/?tavilyApiKey=${{ secrets.TAVILY_API_KEY }}"
22-
allowed: ["*"]
2321
safe-outputs:
2422
add-comment:
2523
max: 1

‎.github/workflows/shared/tavily-mcp.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
mcp-servers:
3+
tavily:
4+
url: "https://mcp.tavily.com/mcp/?tavilyApiKey=${{ secrets.TAVILY_API_KEY }}"
5+
allowed: ["*"]
6+
---

‎docs/src/content/docs/guides/packaging-imports.md‎

Lines changed: 68 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -298,8 +298,9 @@ Imports only a specific section from a markdown file using the section header.
298298

299299
### Frontmatter Merging
300300

301-
- **Only `tools:` frontmatter** is allowed in imported files; other entries give a warning
301+
- **Only `tools:` and `mcp-servers:` frontmatter** is allowed in imported files; other entries give a warning
302302
- **Tool merging**: `allowed:` tools are merged across all imported files
303+
- **MCP server merging**: MCP servers defined in imported files are merged with the main workflow
303304

304305
**Example Tool Merging:**
305306

@@ -326,6 +327,30 @@ tools:
326327

327328
**Result:** Final workflow has `github.allowed: [get_issue, add_issue_comment, update_issue]` and Claude Edit tool.
328329

330+
**Example MCP Server Merging:**
331+
332+
```aw wrap
333+
# Base workflow
334+
---
335+
on: issues
336+
engine: copilot
337+
---
338+
339+
{{#import: shared/tavily-mcp.md}} # Adds Tavily MCP server
340+
```
341+
342+
```aw wrap
343+
# shared/tavily-mcp.md
344+
---
345+
mcp-servers:
346+
tavily:
347+
url: "https://mcp.tavily.com/mcp/?tavilyApiKey=${{ secrets.TAVILY_API_KEY }}"
348+
allowed: ["*"]
349+
---
350+
```
351+
352+
**Result:** Final workflow has the Tavily MCP server configured and available to the AI engine.
353+
329354
### Import Processing During Add
330355

331356
When adding a workflow with the `add` command, local file references in import directives are automatically converted to workflow specifications:
@@ -353,9 +378,12 @@ The `imports:` field in frontmatter provides an alternative way to import files,
353378
imports:
354379
- shared/common-tools.md
355380
- shared/security-setup.md
381+
- shared/tavily-mcp.md
356382
---
357383
```
358384

385+
Imports can include both tool configurations and MCP server definitions from shared files.
386+
359387
### Import Processing
360388

361389
Like `@include` directives, the imports field is processed during the `add` command:
@@ -427,6 +455,44 @@ gh aw update experimental
427455

428456
### Example 3: Using Imports for Modular Workflows
429457

458+
Create a shared MCP server configuration:
459+
460+
```aw wrap
461+
# .github/workflows/shared/tavily-mcp.md
462+
---
463+
mcp-servers:
464+
tavily:
465+
url: "https://mcp.tavily.com/mcp/?tavilyApiKey=${{ secrets.TAVILY_API_KEY }}"
466+
allowed: ["*"]
467+
---
468+
```
469+
470+
Create a workflow that imports the MCP server:
471+
472+
```aw wrap
473+
# .github/workflows/research-agent.md
474+
---
475+
on:
476+
issues:
477+
types: [opened]
478+
479+
imports:
480+
- shared/tavily-mcp.md
481+
482+
tools:
483+
github:
484+
allowed: [add_issue_comment]
485+
---
486+
487+
# Research Agent
488+
489+
Perform web research using Tavily and respond to issues.
490+
```
491+
492+
The final workflow will have both the Tavily MCP server and GitHub tools configured.
493+
494+
### Example 4: Using Imports for Common Tools
495+
430496
Create a base workflow with common tools:
431497

432498
```aw wrap
@@ -466,7 +532,7 @@ Handle new issues with automated responses.
466532

467533
The final workflow will have all three GitHub tools: `get_issue`, `get_pull_request`, and `add_issue_comment`.
468534

469-
### Example 4: Creating Reusable Components
535+
### Example 5: Creating Reusable Components
470536

471537
**Shared security notice:**
472538

‎docs/src/content/docs/reference/include-directives.md‎

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,8 +37,9 @@ Imports only a specific section from a markdown file using the section header.
3737

3838
## Frontmatter Merging
3939

40-
- **Only `tools:` frontmatter** is allowed in imported files, other entries give a warning.
40+
- **Only `tools:` and `mcp-servers:` frontmatter** is allowed in imported files, other entries give a warning.
4141
- **Tool merging**: `allowed:` tools are merged across all imported files
42+
- **MCP server merging**: MCP servers defined in imported files are merged with the main workflow
4243

4344
### Example Tool Merging
4445
```aw wrap
@@ -64,6 +65,30 @@ tools:
6465

6566
**Result**: Final workflow has `github.allowed: [get_issue, add_issue_comment, update_issue]` and Claude Edit tool.
6667

68+
### Example MCP Server Merging
69+
70+
```aw wrap
71+
# Base workflow
72+
---
73+
on: issues
74+
engine: copilot
75+
---
76+
77+
@import shared/tavily-mcp.md # Adds Tavily MCP server
78+
```
79+
80+
```aw wrap
81+
# shared/tavily-mcp.md
82+
---
83+
mcp-servers:
84+
tavily:
85+
url: "https://mcp.tavily.com/mcp/?tavilyApiKey=${{ secrets.TAVILY_API_KEY }}"
86+
allowed: ["*"]
87+
---
88+
```
89+
90+
**Result**: Final workflow has the Tavily MCP server configured and available to the AI engine.
91+
6792
## Import Path Resolution
6893

6994
- **Relative paths**: Resolved relative to the importing file

‎pkg/parser/frontmatter.go‎

Lines changed: 39 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -87,10 +87,11 @@ type FrontmatterResult struct {
8787

8888
// ImportsResult holds the result of processing imports from frontmatter
8989
type ImportsResult struct {
90-
MergedTools string // Merged tools configuration from all imports
91-
MergedEngines []string // Merged engine configurations from all imports
92-
MergedMarkdown string // Merged markdown content from all imports
93-
ImportedFiles []string // List of imported file paths (for manifest)
90+
MergedTools string // Merged tools configuration from all imports
91+
MergedMCPServers string // Merged mcp-servers configuration from all imports
92+
MergedEngines []string // Merged engine configurations from all imports
93+
MergedMarkdown string // Merged markdown content from all imports
94+
ImportedFiles []string // List of imported file paths (for manifest)
9495
}
9596

9697
// ExtractFrontmatterFromContent parses YAML frontmatter from markdown content string
@@ -387,6 +388,7 @@ func ProcessImportsFromFrontmatterWithManifest(frontmatter map[string]any, baseD
387388

388389
// Process each import
389390
var toolsBuilder strings.Builder
391+
var mcpServersBuilder strings.Builder
390392
var markdownBuilder strings.Builder
391393
var engines []string
392394
var processedFiles []string
@@ -451,13 +453,20 @@ func ProcessImportsFromFrontmatterWithManifest(frontmatter map[string]any, baseD
451453
if err == nil && engineContent != "" {
452454
engines = append(engines, engineContent)
453455
}
456+
457+
// Extract mcp-servers from imported file
458+
mcpServersContent, err := extractMCPServersFromContent(string(content))
459+
if err == nil && mcpServersContent != "" && mcpServersContent != "{}" {
460+
mcpServersBuilder.WriteString(mcpServersContent + "\n")
461+
}
454462
}
455463

456464
return &ImportsResult{
457-
MergedTools: toolsBuilder.String(),
458-
MergedEngines: engines,
459-
MergedMarkdown: markdownBuilder.String(),
460-
ImportedFiles: processedFiles,
465+
MergedTools: toolsBuilder.String(),
466+
MergedMCPServers: mcpServersBuilder.String(),
467+
MergedEngines: engines,
468+
MergedMarkdown: markdownBuilder.String(),
469+
ImportedFiles: processedFiles,
461470
}, nil
462471
}
463472

@@ -810,6 +819,28 @@ func extractToolsFromContent(content string) (string, error) {
810819
return strings.TrimSpace(string(toolsJSON)), nil
811820
}
812821

822+
// extractMCPServersFromContent extracts mcp-servers section from frontmatter as JSON string
823+
func extractMCPServersFromContent(content string) (string, error) {
824+
result, err := ExtractFrontmatterFromContent(content)
825+
if err != nil {
826+
return "{}", nil // Return empty object on error
827+
}
828+
829+
// Extract mcp-servers section
830+
mcpServers, exists := result.Frontmatter["mcp-servers"]
831+
if !exists {
832+
return "{}", nil
833+
}
834+
835+
// Convert to JSON string
836+
mcpServersJSON, err := json.Marshal(mcpServers)
837+
if err != nil {
838+
return "{}", nil
839+
}
840+
841+
return strings.TrimSpace(string(mcpServersJSON)), nil
842+
}
843+
813844
// extractEngineFromContent extracts engine section from frontmatter as JSON string
814845
func extractEngineFromContent(content string) (string, error) {
815846
result, err := ExtractFrontmatterFromContent(content)

0 commit comments

Comments
 (0)