Connecting an MCP server has no automated coverage. The two acceptance suites that drive that journey now skip in every environment that runs them, and the check reports success.
What happens
web/oss/tests/playwright/acceptance/settings/mcp-connect.ts:95-105 and web/oss/tests/playwright/acceptance/playground/mcp-agent-config.ts:341-351 both start with:
const {reachable, reason} = await ensureMockMcpUpstream()
test.skip(!reachable, reason)
ensureMockMcpUpstream (web/oss/tests/playwright/acceptance/utils/mcpConnections.ts:115-135) dials the mock MCP upstream once per worker and reports whether the journey can run. The hosted preview is the only environment that runs web acceptance, and it never brings up the gateway mocks, so the probe always fails there and all seven cases skip.
The skip itself was the right call when it was made. These suites were red on every unrelated pull request, which is worse than no signal. The consequence is that the MCP connect journey is now covered only by a compose or local stack started with AGENTA_GATEWAYS_MOCKS_ENABLED=true, which no automated run does. A green acceptance check says nothing about it.
Related gap
Nothing runs on a push to a release branch either, so the commits that actually ship are never checked by these suites or any other. That half is tracked in #6967. Together it means MCP connect can break on a release branch and no automated run would notice.
Proposed fix
Run the mock MCP upstream in the environment that runs the acceptance suites, so the probe succeeds and the cases execute:
- Add the mock upstream to the compose stack used for acceptance runs, behind
AGENTA_GATEWAYS_MOCKS_ENABLED, and publish it at an address the suites can dial.
- Point
AGENTA_MOCK_MCP_GATEWAY_PUBLISHED_URL at it, which the helper already reads.
- Once the cases run somewhere, make the skip loud rather than silent: report the skipped count on the check, or fail when these particular suites skip in an environment that is supposed to run them.
Step 3 matters on its own. A suite that skips everywhere and reports success is indistinguishable from a suite that passes, which is how this went unnoticed.
Related: #6913 and #6916 record other ways these acceptance suites give less signal than their result suggests.
Connecting an MCP server has no automated coverage. The two acceptance suites that drive that journey now skip in every environment that runs them, and the check reports success.
What happens
web/oss/tests/playwright/acceptance/settings/mcp-connect.ts:95-105andweb/oss/tests/playwright/acceptance/playground/mcp-agent-config.ts:341-351both start with:ensureMockMcpUpstream(web/oss/tests/playwright/acceptance/utils/mcpConnections.ts:115-135) dials the mock MCP upstream once per worker and reports whether the journey can run. The hosted preview is the only environment that runs web acceptance, and it never brings up the gateway mocks, so the probe always fails there and all seven cases skip.The skip itself was the right call when it was made. These suites were red on every unrelated pull request, which is worse than no signal. The consequence is that the MCP connect journey is now covered only by a compose or local stack started with
AGENTA_GATEWAYS_MOCKS_ENABLED=true, which no automated run does. A green acceptance check says nothing about it.Related gap
Nothing runs on a push to a release branch either, so the commits that actually ship are never checked by these suites or any other. That half is tracked in #6967. Together it means MCP connect can break on a release branch and no automated run would notice.
Proposed fix
Run the mock MCP upstream in the environment that runs the acceptance suites, so the probe succeeds and the cases execute:
AGENTA_GATEWAYS_MOCKS_ENABLED, and publish it at an address the suites can dial.AGENTA_MOCK_MCP_GATEWAY_PUBLISHED_URLat it, which the helper already reads.Step 3 matters on its own. A suite that skips everywhere and reports success is indistinguishable from a suite that passes, which is how this went unnoticed.
Related: #6913 and #6916 record other ways these acceptance suites give less signal than their result suggests.