Skip to content

(chore) MCP connect has no automated coverage since both acceptance suites skip everywhere #7045

Description

@mmabrouk

Connecting an MCP server has no automated coverage. The two acceptance suites that drive that journey now skip in every environment that runs them, and the check reports success.

What happens

web/oss/tests/playwright/acceptance/settings/mcp-connect.ts:95-105 and web/oss/tests/playwright/acceptance/playground/mcp-agent-config.ts:341-351 both start with:

const {reachable, reason} = await ensureMockMcpUpstream()
test.skip(!reachable, reason)

ensureMockMcpUpstream (web/oss/tests/playwright/acceptance/utils/mcpConnections.ts:115-135) dials the mock MCP upstream once per worker and reports whether the journey can run. The hosted preview is the only environment that runs web acceptance, and it never brings up the gateway mocks, so the probe always fails there and all seven cases skip.

The skip itself was the right call when it was made. These suites were red on every unrelated pull request, which is worse than no signal. The consequence is that the MCP connect journey is now covered only by a compose or local stack started with AGENTA_GATEWAYS_MOCKS_ENABLED=true, which no automated run does. A green acceptance check says nothing about it.

Related gap

Nothing runs on a push to a release branch either, so the commits that actually ship are never checked by these suites or any other. That half is tracked in #6967. Together it means MCP connect can break on a release branch and no automated run would notice.

Proposed fix

Run the mock MCP upstream in the environment that runs the acceptance suites, so the probe succeeds and the cases execute:

  1. Add the mock upstream to the compose stack used for acceptance runs, behind AGENTA_GATEWAYS_MOCKS_ENABLED, and publish it at an address the suites can dial.
  2. Point AGENTA_MOCK_MCP_GATEWAY_PUBLISHED_URL at it, which the helper already reads.
  3. Once the cases run somewhere, make the skip loud rather than silent: report the skipped count on the check, or fail when these particular suites skip in an environment that is supposed to run them.

Step 3 matters on its own. A suite that skips everywhere and reports success is indistinguishable from a suite that passes, which is how this went unnoticed.

Related: #6913 and #6916 record other ways these acceptance suites give less signal than their result suggests.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions