Skip to content

Repository files navigation

telemetry.dev logo

telemetry.dev

LLM observability for AI applications and agents


npm · PyPI · MIT License

Trace model calls, tool steps, and agent runs. Find slow requests, debug errors, and monitor token usage and cost.

Start tracing · Documentation · Integrations

telemetry.dev connects your AI application to a shared view of traces, logs, and metrics. This repository contains the official TypeScript and Python SDKs and integrations, built on OpenTelemetry.

Quick start

Make a project API key at telemetry.dev. Set the API keys for telemetry.dev and your model provider:

export TELEMETRY_DEV_API_KEY="your-telemetry-dev-api-key"
export OPENAI_API_KEY="your-openai-api-key"

TypeScript

Use Node.js 24 or newer. Install the SDK and the OpenAI integration:

npm install @telemetry-dev/sdk @telemetry-dev/openai openai

Save this as trace.ts:

import OpenAI from "openai";
import { init, shutdown } from "@telemetry-dev/sdk";
import { wrapOpenAI } from "@telemetry-dev/openai";

init({ serviceName: "my-ai-app" });
const openai = wrapOpenAI(new OpenAI());

try {
  const response = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [{ role: "user", content: "What is OpenTelemetry?" }],
  });
  console.log(response.choices[0]?.message.content);
} finally {
  await shutdown();
}

Run the request:

node trace.ts

Open your project in telemetry.dev to inspect the trace. The integration records the model call, reported token usage, duration, and errors.

Python quick start

Use Python 3.10 or newer. Install the OpenAI integration:

pip install telemetry-dev-openai

Use the same API keys as the TypeScript example. Save this as trace.py:

import telemetry_dev
from openai import OpenAI
from telemetry_dev_openai import wrap_openai

telemetry_dev.init(service_name="my-ai-app")
client = wrap_openai(OpenAI())

try:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "What is OpenTelemetry?"}],
    )
    print(response.choices[0].message.content)
finally:
    telemetry_dev.shutdown()

Run the request:

python trace.py

The SDKs capture inputs and outputs by default. To disable input and output capture, set captureInput: false and captureOutput: false in TypeScript init(). For Python, use capture_input=False and capture_output=False. Disabling output capture omits message content; response IDs and models, finish reasons, usage, timing, and errors may still be recorded when providers supply them. For streamed OpenAI chat, usage requires provider support or the documented stream-usage option in the OpenAI integration. Capture flags do not gate stop sequences, tool descriptions and definitions (gen_ai.tool.description, gen_ai.tool.definitions), caller-supplied metadata or raw attributes, or exception messages and stack traces; redact those separately when needed. OpenAI streamed chat and Responses reconstruction, and Anthropic stream reconstruction, retain up to 48 KiB and 1,000 items; OpenAI transcription text retains up to 64 KiB with no item limit. These are fixed resource bounds: the configured attribute-length limit is applied by the core to the exported attribute after masking, not to stream retention. A truncated or otherwise incomplete stream sets telemetry.dev.capture.truncated; when a mask is configured, incomplete reconstructed output is omitted because the mask cannot inspect the complete value.

Features

  • Agent traces. Connect model calls, tool steps, and custom spans in one trace.
  • Usage and cost. Inspect reported token usage, cost, latency, and errors in telemetry.dev.
  • Provider integrations. Add tracing to your existing model client without a separate request API.
  • Session context. Group activity by user and session, and pass trace context between services.
  • Capture controls. Disable SDK input and output capture, or mask captured content before export.
  • OpenTelemetry. Send traces, logs, and metrics through standard OTLP/HTTP.

Integrations

Each integration links to its installation instructions and API coverage.

Integration TypeScript Python
Core SDK SDK SDK
Existing OpenTelemetry setup Span processor Span processor
OpenAI OpenAI OpenAI
Anthropic Anthropic Anthropic
Amazon Bedrock Bedrock Bedrock
Google GenAI Google GenAI Google GenAI
OpenRouter OpenRouter OpenRouter
Vercel AI SDK AI SDK
TanStack AI TanStack AI
LiteLLM LiteLLM
MCP MCP
Eve Eve
Cursor Cursor
OpenCode OpenCode
Oh My Pi Oh My Pi
Pi Pi

Other languages can use an existing OTLP/HTTP exporter. The documentation describes endpoint configuration and authentication.

Development

Local development uses Node.js 24, pnpm 11.25.0, Vite+, uv, and Python 3.10 or newer.

pnpm install --frozen-lockfile
pnpm run check
pnpm run test
pnpm run build

pnpm run py:sync
pnpm run py:check
pnpm run py:test
pnpm run py:test:bounds
pnpm run py:build

py:test:bounds runs each Python package's tests against the lowest and the highest third-party dependency versions its declared ranges allow. It always uses the local telemetry-dev checkout, not the lowest published telemetry-dev a provider package accepts.

CI runs dependency, code, test, build, and release-safety checks for TypeScript and Python. The SDK conformance guide defines the shared telemetry format and behavior.

Release maintenance

Releases

The Release workflow uses Release Please to propose version bumps on main. Merging a release PR lets Release Please create component releases such as sdk-v0.1.3 or python-v0.2.3. Each published release builds and publishes only its matching package. Build jobs have no publishing credentials; separate publish jobs use registry OIDC trusted publishing, not long-lived npm or PyPI tokens.

The manifest retains versions already published from the original repository. bootstrap-sha points to the initial SDK import so that import's feat commit does not trigger new versions. Do not create releases to republish these baseline versions: wait for subsequent releasable changes. No new version is required merely to transfer publishing to this repository.

Required setup

The workflow is restored, but publishing is not fully configured until the following handoff is complete:

  • Provide the repository Actions secret RELEASE_PLEASE_TOKEN, authorized for telemetry-dev/telemetry with contents, issues, and pull-request write permissions. The original repository's secret cannot be read or copied from GitHub; its owner must supply or replace it. The workflow deliberately fails when this secret is missing. Do not substitute GITHUB_TOKEN: releases created with it do not trigger the downstream release workflow.
  • Configure every npm package's GitHub Actions trusted publisher for owner telemetry-dev, repository telemetry, and workflow filename release.yml, without an environment name. npm publishing runs on GitHub-hosted runners and includes provenance for this public repository.
  • Configure every PyPI project's GitHub Actions trusted publisher for owner telemetry-dev, repository telemetry, and workflow filename release.yml, and environment pypi. The repository's pypi environment permits branch main for retries and tags matching python*-v*; it has no required reviewers. These branch/tag restrictions are not a manual approval gate.
  • Require successful CI before merging release PRs. Verify all trusted-publisher settings and the secret before expecting an end-to-end publish to succeed.

Validation and retries

Run the release safety suite locally with Bash, Git, jq, Python 3, tar, and OpenSSL; the registry commands in these tests are mocked and publish nothing:

bash .github/scripts/release_test.sh
shellcheck .github/scripts/*.sh
actionlint

To retry an existing published release, run the Release workflow from main and set release_tag to that release's full tag. This is a real publication attempt, not a dry run. The tag must be an ancestor of the selected main commit. Every configured SDK directory, shared build scripts, lockfiles, release configuration, and root build configuration must be unchanged from the tag; only packageManager may differ within root package.json. Workflow-only recovery changes are allowed. Source-changing fixes need a new release, not a retry of an old version.

Artifact names and versions are checked before upload. npm retries skip an existing version only when its SHA-512 integrity matches the local tarball, and publication waits for published @telemetry-dev/* dependencies and peer dependencies to satisfy their declared ranges. PyPI retries use uv publish --check-url to check existing distributions. An integrity mismatch must be investigated, not bypassed or overwritten.

License

This project uses the MIT License.

Links

About

Trace AI applications and agents, and monitor token usage, cost, latency, and errors with OpenTelemetry.

Resources

Stars

32 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages