This directory contains DataDog dashboard and monitor configurations for Thread performance monitoring and constitutional compliance validation.

datadog/
├── dashboards/
│   └── thread-performance-monitoring.json   # Main performance dashboard
└── README.md                                # This file

Purpose: Monitor Thread's constitutional compliance and operational performance

Key Features:

• Constitutional compliance gauges (cache hit rate >90%, query latency <50ms)
• Performance metrics (fingerprint computation, query execution)
• Throughput monitoring (file processing, data throughput, batch operations)
• Cache operations tracking (hits, misses, evictions)
• Error rate monitoring

Metrics Used:

• thread.cache_hit_rate_percent - Cache hit rate percentage
• thread.query_avg_duration_seconds - Average query latency
• thread.fingerprint_avg_duration_seconds - Fingerprint computation time
• thread.files_processed_total - Total files processed
• thread.bytes_processed_total - Total bytes processed
• thread.batches_processed_total - Total batches processed
• thread.cache_hits_total - Total cache hits
• thread.cache_misses_total - Total cache misses
• thread.cache_evictions_total - Total cache evictions
• thread.query_errors_total - Total query errors
• thread.query_error_rate_percent - Query error rate percentage

See docs/operations/DASHBOARD_DEPLOYMENT.md for detailed deployment instructions.

Via UI:

1. DataDog UI → Dashboards → New Dashboard → Import JSON
2. Paste contents of dashboards/thread-performance-monitoring.json
3. Save dashboard

Via API:

DD_API_KEY="your-api-key"
DD_APP_KEY="your-app-key"

curl -X POST "https://api.datadoghq.com/api/v1/dashboard" \
  -H "DD-API-KEY: ${DD_API_KEY}" \
  -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
  -H "Content-Type: application/json" \
  -d @datadog/dashboards/thread-performance-monitoring.json

Via Terraform:

resource "datadog_dashboard_json" "thread_performance" {
  dashboard = file("${path.module}/datadog/dashboards/thread-performance-monitoring.json")
}

Configure the DataDog Agent to scrape Thread's Prometheus metrics endpoint:

# /etc/datadog-agent/datadog.yaml
prometheus_scrape:
  enabled: true
  configs:
    - configurations:
      - timeout: 5
        prometheus_url: "http://thread-service:8080/metrics"
        namespace: "thread"
        metrics:
          - "thread_*"

# Check if DataDog is collecting Thread metrics
datadog-agent status | grep thread

# Query metrics via DataDog API
curl -X GET "https://api.datadoghq.com/api/v1/metrics?from=$(date -d '1 hour ago' +%s)&metric=thread.cache_hit_rate_percent" \
  -H "DD-API-KEY: ${DD_API_KEY}" \
  -H "DD-APPLICATION-KEY: ${DD_APP_KEY}"

Constitutional Compliance Alerts:

1. Cache Hit Rate Below 90%:

   {
     "name": "Thread Cache Hit Rate Below Constitutional Minimum",
     "type": "metric alert",
     "query": "avg(last_5m):avg:thread.cache_hit_rate_percent{*} < 90",
     "message": "Cache hit rate is below 90% constitutional requirement",
     "tags": ["team:thread", "priority:high", "constitutional-compliance"]
   }

2. Query Latency Exceeds 50ms:

   {
     "name": "Thread Query Latency Exceeds Constitutional Maximum",
     "type": "metric alert",
     "query": "avg(last_5m):avg:thread.query_avg_duration_seconds{*} * 1000 > 50",
     "message": "Query latency exceeds 50ms constitutional requirement",
     "tags": ["team:thread", "priority:high", "constitutional-compliance"]
   }

Operational Alerts:

3. High Error Rate:

   {
     "name": "Thread Query Error Rate Too High",
     "type": "metric alert",
     "query": "avg(last_5m):avg:thread.query_error_rate_percent{*} > 1",
     "message": "Query error rate exceeds 1%",
     "tags": ["team:thread", "priority:medium"]
   }

4. Cache Eviction Storm:

   {
     "name": "Thread High Cache Eviction Rate",
     "type": "metric alert",
     "query": "avg(last_5m):per_second(avg:thread.cache_evictions_total{*}) > 100",
     "message": "Cache eviction rate indicates memory pressure",
     "tags": ["team:thread", "priority:low"]
   }

1. Edit the dashboard JSON file
2. Add new widget definition to widgets array
3. Use Thread metrics (thread.*)
4. Redeploy dashboard

The dashboard includes a template variable for environment filtering:

"template_variables": [
  {
    "name": "environment",
    "default": "production",
    "prefix": "environment",
    "available_values": ["production", "staging", "development"]
  }
]

To use in queries: thread.cache_hit_rate_percent{$environment}

Thread also provides Grafana dashboards in grafana/dashboards/.

Key Differences:

• Grafana uses Prometheus metrics directly (underscores: thread_*)
• DataDog converts metric names (dots: thread.*)
• Both monitor the same underlying metrics from PerformanceMetrics

Choose Based On:

• Grafana: If you already have Prometheus infrastructure
• DataDog: If you use DataDog for other services
• Both: For redundancy and cross-validation

1. Check Agent Status:

   sudo datadog-agent status

2. Verify Prometheus Integration:

   sudo datadog-agent check prometheus -t

3. Check Metrics Endpoint:

   curl http://thread-service:8080/metrics | grep thread_cache_hit_rate_percent

1. Verify Metric Collection:

   # DataDog Metrics Explorer
   # Query: thread.cache_hit_rate_percent

2. Check Conversion:

   • Prometheus: thread_cache_hit_rate_percent (with underscore)
   • DataDog: thread.cache_hit_rate_percent (with dot)
   • DataDog Agent auto-converts underscores to dots

1. Validate JSON:

   jq '.' datadog/dashboards/thread-performance-monitoring.json

2. Check Permissions:

   • Ensure API and App keys have dashboard creation permissions
   • Verify user role includes dashboard management

• Deployment Guide: docs/operations/DASHBOARD_DEPLOYMENT.md
• Performance Metrics: crates/flow/src/monitoring/performance.rs
• Constitutional Requirements: .specify/memory/constitution.md
• Monitoring Overview: docs/operations/MONITORING.md

Version: 1.0.0 Last Updated: 2026-01-28 Author: Thread Operations Team (via Claude Sonnet 4.5)