mcp-observability (mcp-stats)

An infrastructure-only Terraform project that aggregates usage telemetry — sessions, unique clients, tool-call data — across the whole fleet of hosted MCP (Model Context Protocol) servers, each of which runs as its own AWS Lambda behind API Gateway.

It has its own git repo and its own Terraform state. It has zero code coupling to the MCP servers — it only reads the CloudWatch log groups they already produce.

All MCPs run in a single AWS account and region (us-west-2).

No new infrastructure sits in any request path. Everything here reads log groups the MCPs already emit:

1. Tag-based discovery (discovery.tf) — finds every MCP log group via the Resource Groups Tagging API, filtered on Project = mcp-server.
2. Account-wide CloudWatch dashboard (dashboard.tf) — one dashboard spanning all discovered MCPs: sessions/day (fleet and per-MCP), unique clients/day, tool popularity by MCP, per-MCP request volume, client family, top source IPs.
3. Saved Logs Insights queries (queries.tf) — the cross-MCP versions of each MCP repo's per-MCP "Tier 1 usage tracking" queries, plus an error-triage set that separates real server faults from client-fault rejections.
4. Fleet MCP-route 4xx alarms (mcp_4xx_alarms.tf) — for every discovered access log group, a metric filter counting 4xx responses on POST /mcp only, an alarm on it (>= 100 per 5 min by default), and one shared SNS topic mcp-fleet-alarms they all notify. This is the signal the per-repo apigw-4xx-probing alarms were after, minus the edge 403s that web-vulnerability scanners generate walking /.env-style paths — those never reach /mcp, so they no longer page. Metric filters have no backfill; verify a pattern with aws logs test-metric-filter.

Each MCP repo tags both its Lambda log group (/aws/lambda/<mcp>-<env>) and its API Gateway access log group (/aws/apigateway/<mcp>-<env>-access) with Project = mcp-server. This project reads that tag through the Resource Groups Tagging API — note the aws_cloudwatch_log_groups data source filters by name prefix only, not by tag, so tag discovery must go through the tagging API.

The fleet runs both staging and prod under the same tag, so discovery.tf scopes to one environment via var.environment (default prod) by matching the trailing -<env> name segment. Set environment = "" to include every environment.

discovery.tf is the single reusable discovery layer. Phase 2 (below) will consume the exact same local.mcp_lambda_log_groups / local.mcp_apigw_log_groups lists — never re-derive the log group list elsewhere.

All MCP repos carry the Project = mcp-server tag. With the default environment = prod, discovery resolves to:

That is 14 MCPs, matching discovered_mcp_count. If this table and that output disagree, the output is right — it is what discovery actually resolved on the last apply.

The seven servers added 2026-07-13 (Anchorage Parcels, Anchorage eCode, Audubon IBA, both San Diego GIS servers, ESRI UC 2026, ESRI Living Atlas), Anchorage Checkbook, and Alaska Geoportal (prod since 2026-09-15) all run the shared Python core/ codebase: their Lambda logs carry the full jsonrpc_* fields and their access logs emit sourceIp + userAgent, so every widget and saved query covers them with no schema gaps.

Known gaps / variances:

• Discovery is eventually consistent. The Resource Groups Tagging API indexes tags asynchronously, so get-resources can return slightly different result sets between calls — and a freshly tagged (or freshly destroyed) log group may take a while to appear or drop out. Each terraform plan/apply re-reads the API, so the discovered list can momentarily flap if a run catches an in-between view; the next run self-corrects. This is inherent to tag-based discovery, not a bug in this project.
• Access-log schema is not uniform. eBird and Anchorage GIS emit sourceIp + userAgent; Boston and Census emit ip and no userAgent. Queries normalise with coalesce(sourceIp, ip); the userAgent-based "unique client" proxy degrades to IP-only for MCPs that omit it.
• Lambda logs are retained for only 14 days by the MCP repos (access logs: 30 days on most, 14 on Boston and Census), so every log-backed widget and saved query goes blind past two weeks. The dashboard's Row 7 reads the AWS/Lambda Invocations metric instead (15 months of history) for month-over-month shape; for anything older than 14 days that needs jsonrpc_* fields, there is no data — that is what Phase 2 is for.
• claude.ai's connector refresh looks like usage but is not. claude.ai re-runs initialize + tools/list against every connected MCP about twice a day from Anthropic egress (160.79.106.x, user agent Claude-User, clientInfo Anthropic/ClaudeAI) without calling a tool. At current volume that is most of the fleet's sessions, initializes and "unique clients". Judge adoption from the Row 5 tools/call-only widgets and the real-* saved queries; the rest measure connectivity.
• Newer-core MCPs reject MCP-Protocol-Version: 2026-07-28 with a 400 (core from 2026-08-24 on; as of 2026-10-21 that is eBird, Anchorage eCode/Checkbook, both San Diego servers and Alaska Geoportal. Anchorage GIS deployed the dual-era core on 2026-09-17 and Anchorage Parcels on 2026-10-21 (dual-era + the -32022 id-echo follow-up, GIS 5520d7e); both now serve 2026-07-28 — their counts are zero, and claude.ai sessions against them open with server/discover and carry NO mcp_session_id, so the session-id-based widgets under-count them; the client-family widget and client-family-breakdown query count session opens in both eras). The first request of every claude.ai / claude-code session gets the 400, the client retries with an older version about a second later and succeeds, so nothing is user-visible — but it is ~15% of all POST /mcp, inflates the fleet 4xx metric, and is the top row of the protocol-rejections widget. The fix is in the shared core's supported-version list; the saved query mcp-fleet/errors/protocol-version-400s-per-day shows each MCP dropping to zero as the fix is deployed, and can be deleted once they all have.
• Census runs a different codebase (Node.js, not the shared Python core/). If its Lambda logs do not carry identical jsonrpc_* field names, the Lambda-log widgets simply show no Census rows — non-fatal.
• Staging groups exist for several MCPs (ebird-mcp-staging, boston-ckan-mcp-staging, …) and are excluded by the default prod scope.
• Per-MCP cost attribution is forward-only. The Project tag was activated as a cost-allocation tag on 2026-05-30, so Cost Explorer can now break Lambda / API Gateway / CloudWatch spend down per MCP — but only for usage from that date forward; AWS does not backfill tag-based cost data, so earlier spend is unattributable. Note also that the AWS account is shared with unrelated infrastructure (RDS, VPC, WAF, …), so account-level totals are not a proxy for fleet cost. The fleet's own serverless footprint is tiny — Lambda + API Gateway run at fractions of a cent/week at current traffic; CloudWatch logs are the only material line item.

# 1. Bootstrap the (separate) S3 + DynamoDB backend.
./scripts/setup-backend.sh
cp terraform/aws/backend.tf.example terraform/aws/backend.tf   # if not auto-written (gitignored)

# 2. Plan and apply.
cd terraform/aws
terraform init
terraform plan
terraform apply

terraform output dashboard_url prints the console link once applied.

The fleet alarm email lives in a gitignored, auto-loaded terraform/aws/*.auto.tfvars (fleet_alarm_email = "...") so it stays out of the public repo. The first apply creates an SNS email subscription that sits in PendingConfirmation until the confirmation link in the email is clicked; until then the alarms fire but deliver nothing. Pass fleet_alarm_sns_topic_arn instead to reuse an existing topic — the live deployment does this, pointing at anchorage-gis-mcp-prod-alarms (owned by the anchorage-gis repo, already confirmed) because the confirmation email for a fresh topic never arrived. If that repo's topic is ever destroyed, clear the variable and re-apply to fall back to a project-owned topic.

• Separate Terraform state from every MCP repo — see scripts/setup-backend.sh and backend.tf.example.
• Additive only. This project does not modify or remove any MCP's existing per-MCP dashboard.
• The only change this project's design ever pushes into an MCP repo is the Project = mcp-server tag on its log groups — and that rollout is already complete across the fleet, so Phase 1 touches no MCP repo at all.

Later this project will add CloudWatch subscription filters on each discovered log group → Kinesis Firehose → S3 → Athena, for durable history and SQL. It is intentionally not built yet. When it is, it consumes the same discovery.tf locals the dashboard and queries already use.