For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Metrics

Monitoring your MAX deployment helps you identify latency bottlenecks, optimize hardware utilization, and ensure reliable inference at scale. MAX provides real-time operational insights through Prometheus metrics and OpenTelemetry (OTLP) so you can track system health from request entry to token generation.

How to use the /metrics endpoint

MAX provides a /metrics endpoint that returns metrics in Prometheus text format. You can scrape this endpoint with Prometheus or another metrics collector.

The endpoint is available when you run the max serve command and when you use the MAX container.

By default, the endpoint listens on port 8001. You can change this with the MAX_SERVE_METRICS_ENDPOINT_PORT environment variable.

To verify the endpoint is working, send a request with curl:

curl http://localhost:8001/metrics

This returns all metrics in Prometheus text format. You can filter for specific metrics:

curl -s http://localhost:8001/metrics | grep maxserve_time_to_first_token

To scrape metrics automatically, add a job to your Prometheus configuration:

scrape_configs:
  - job_name: max-serve
    scrape_interval: 15s
    static_configs:
      - targets: ["localhost:8001"]

If you changed the metrics port with MAX_SERVE_METRICS_ENDPOINT_PORT, update the target accordingly.

Reference

Review all available metrics grouped by type.

Request latency

The following metrics track end-to-end and per-phase request latency.

Metric	Type	Description
maxserve_request_time_milliseconds	Histogram	Total time spent handling a request, measured from the HTTP middleware receiving it to the response completing: request parsing, validation, queueing, model execution and response generation.
maxserve_input_processing_time_milliseconds	Histogram	Input processing time (IPT).
maxserve_output_processing_time_milliseconds	Histogram	Output generation time (OGT).
maxserve_time_to_first_token_milliseconds	Histogram	Time to first token (TTFT). Measured from when the server received the request, so it includes request parsing, validation, media resolution, tokenization, queueing, and prefill.
maxserve_time_to_first_token_by_prompt_length_milliseconds	Histogram	Time to first token as maxserve_time_to_first_token_milliseconds measures it, split by a prompt_length label that buckets the request's prompt tokens in units of 1024: 0-8k, 8k-32k, 32k-64k, 64k-128k, 128k-256k or 256k-plus. Compares TTFT across deployments at matched prompt length.
maxserve_itl_milliseconds	Histogram	Inter-token latency.
maxserve_time_per_output_token_milliseconds	Histogram	Mean decode-phase latency per generated token (TPOT). Emitted once per request as decode_time / (num_generated_tokens - 1). Excludes the first token and prefill/TTFT.

Token counts

The following metrics track cumulative and per-request token volumes, and per-request prefix cache coverage.

Metric	Type	Description
maxserve_num_input_tokens_total	Counter	Cumulative input tokens processed.
maxserve_num_output_tokens_total	Counter	Cumulative output tokens generated.
maxserve_input_tokens_per_request_tokens	Histogram	Distribution of input tokens per request. Chat requests carry a turn label: first when the conversation has no assistant message yet, later when it resends earlier turns. Completion requests carry none.
maxserve_prefix_coverage_per_request_percent	Histogram	Per-request prefix cache coverage (cached prefix tokens / prompt tokens) for chat requests, recorded when the request ends. Carries a turn label: first when the conversation has no assistant message yet, later when it resends earlier turns. maxserve_cache_request_prefix_coverage_percent records the same ratio for every request from the scheduler, without the label.
maxserve_output_tokens_per_request_tokens	Histogram	Distribution of output tokens per request. Chat requests carry a turn label: first when the conversation has no assistant message yet, later when it resends earlier turns. Completion requests carry none.

Request counts and state

The following metrics track request volume and queue depth.

Metric	Type	Description
maxserve_request_count_total	Counter	Total API requests since server start. Liveness and observability endpoints (/health, /version, /ping and /metrics) are excluded.
maxserve_request_client_disconnect_cancellations_total	Counter	Requests whose client disconnected before the server finished responding, split by phase. before_response covers the chat and text completion routes, which watch for a disconnect while a request waits for its result; maxserve_request_count_total also records these as code 499. during_response is a disconnect after a streamed response started and before its last chunk, which maxserve_request_count_total records under the response's status code, so it is visible only here. Both phases are exported at zero from startup.
maxserve_num_requests_queued	Gauge	Current depth of the scheduler's CE/prefill queue, sampled once per scheduler iteration.
maxserve_num_requests_running	Gauge	Requests currently being processed.
maxserve_num_requests_awaiting_admission	Gauge	Requests received by the API server but not yet handed off to the model worker. A persistently high value indicates a backlog in the API server rather than the scheduler.
maxserve_requests_awaiting_admission	Histogram	Distribution of the ingress backlog (requests accepted by the API server but not yet handed off to the model worker), sampled periodically. Captures the p50/p99 tail over time.

Response buffering

The following metrics track the egress backlog between the model worker and the streaming layer.

Metric	Type	Description
maxserve_num_responses_buffered	Gauge	Model-worker responses received by the API server but not yet consumed by the streaming layer (sum of per-request output-queue depths), sampled periodically.
maxserve_responses_buffered	Histogram	Distribution of the egress backlog, sampled periodically. Captures the p50/p99 tail that a scrape-interval gauge sample would miss.
maxserve_response_queue_time_milliseconds	Histogram	Time a model-worker response waits in the per-request output queue before the streaming layer consumes it.

Batch scheduler

The following metrics track batch composition and per-batch throughput.

Metric	Type	Description
maxserve_batch_size	Histogram	Distribution of batch sizes (number of requests), labeled by batch_type (CE prefill or TG decode).
maxserve_batch_execution_time_milliseconds	Histogram	Host executor service time per completed batch.
maxserve_scheduler_batch_step_time_milliseconds	Histogram	Whole scheduler step including execution, request advancement, release and response publication; excludes batch construction.
maxserve_batch_input_tokens	Histogram	Distribution of input tokens per scheduler batch.
maxserve_batch_context_tokens	Histogram	Distribution of accumulated context tokens per scheduler batch.
maxserve_batch_creation_time_milliseconds	Histogram	Distribution of scheduler batch creation time.
maxserve_batch_prompt_throughput_tokens_per_second	Histogram	Per-batch prompt-side throughput per second.
maxserve_batch_generation_throughput_tokens_per_second	Histogram	Per-batch generation-side throughput per second.
maxserve_batch_terminated_reqs	Histogram	Distribution of requests terminated per scheduler batch.
maxserve_batch_pending_reqs	Histogram	Distribution of requests pending in the queue, sampled once per scheduler batch.

Batch execution time is a host executor service interval, not GPU kernel time. Without overlap it measures the call to pipeline.execute(), excluding batch construction, scheduler advancement, release, and response publication. With overlap it is the observed completion time minus the later of the batch's enqueue time and the previous successful batch's observed completion time.

Both modes include executor preparation, sampling, and synchronization. Overlap can also include time between the forward finishing and the host observing its completion, including other batches' scheduling and advancement. Successful completions record once, including final drains; failed launches and completions do not contribute a successful execution-time sample.

The prompt throughput numerator is all active input tokens, including decode rows in mixed batches. Generation throughput counts returned output tokens before scheduler filtering, including prefill outputs and accepted plus bonus speculative outputs. Both rates use this execution interval, and overlap labels and counts describe the completed batch. A completion with zero duration records time but omits rates.

This timing change retains MAX's existing metric names and CE / TG labels. Mach retains its own names and prefill / decode / mixed labels; no shared name or batch-label migration is part of this change.

maxserve.scheduler.batch_step_time preserves the former full scheduler-step interval as a separate diagnostic; it is not a throughput denominator.

KV cache

The following metrics track KV cache utilization, prefix cache hit rates, and KV movement across memory tiers and replicas. Device figures are in blocks, the unit the cache allocates in; the connector's host and disk tiers are in bytes, since those are byte budgets whose block width need not match the device's.

Metric	Type	Description
maxserve_cache_num_used_blocks	Gauge	Number of KV cache blocks currently in use.
maxserve_cache_num_total_blocks	Gauge	Total number of KV cache blocks.
maxserve_cache_jenga_small_blocks	Gauge	A Jenga KV leaf's small blocks on one replica, by state: used by a request, evictable (unreferenced but still in the prefix cache), or empty to allocate without evicting. The leaf's remaining small blocks are bytes other leaves hold. Sampled at the scheduler's log interval.
maxserve_cache_jenga_huge_blocks	Gauge	Jenga huge blocks on one replica, by leaf and state: used (a request holds small blocks in it) or evictable (no request holds it, but it is still typed to the leaf), plus leaf=none, state=empty for those no leaf has claimed. Sums to the replica's allocatable huge blocks. Sampled at the scheduler's log interval.
maxserve_cache_request_prefix_coverage_percent	Histogram	Per-request prefix cache coverage (cached prefix tokens / prompt tokens). For the token-weighted cache hit rate, derive it from maxserve_cache_hits_tokens_total and maxserve_cache_misses_tokens_total.
maxserve_cache_preemption_count_total	Counter	Total number of preemptions.
maxserve_cache_hits_tokens_total	Counter	Cumulative KV cache hit tokens (prompt tokens served from the prefix cache). Tagged with a tier label (g0 for the on-device prefix cache, external for the KV connector). The per-tier series sum to the untagged total.
maxserve_cache_misses_tokens_total	Counter	Cumulative KV cache miss tokens (prompt tokens actually prefilled by the model).
maxserve_cache_used_kv_pct_percent	Histogram	Percentage of device KV cache blocks in use (0-100%), sampled once per scheduler batch.
maxserve_cache_used_host_kv_pct_percent	Histogram	Percentage of the KV connector's host tier bytes in use (0-100%), sampled once per scheduler batch when a connector with a host tier is configured.
maxserve_cache_device_blocks_served_blocks_total	Counter	Cumulative KV blocks served directly from the local device prefix cache.
maxserve_cache_h2d_bytes_copied_bytes_total	Counter	Cumulative KV bytes copied from the connector's host tier to device. Rate this to see the PCIe bandwidth the host tier consumes.
maxserve_cache_cross_replica_blocks_copied_blocks_total	Counter	Cumulative KV blocks copied device-to-device across data-parallel replicas to reuse a prefix-cache hit on another replica.
maxserve_cache_cross_replica_bytes_copied_bytes_total	Counter	Cumulative bytes moved by device-to-device KV copies across data-parallel replicas.
maxserve_cache_d2h_bytes_copied_bytes_total	Counter	Cumulative KV bytes copied from device to the connector's host tier.
maxserve_cache_disk_bytes_read_bytes_total	Counter	Cumulative KV bytes read from the disk cache tier.
maxserve_cache_disk_bytes_written_bytes_total	Counter	Cumulative KV bytes written to the disk cache tier.
maxserve_cache_used_disk_kv_pct_percent	Histogram	Percentage of the KV connector's disk tier bytes in use (0-100%), sampled once per scheduler batch when the connector has a disk tier.
maxserve_cache_connector_load_failures_loads_total	Counter	Cumulative KV connector loads whose copy failed. Each is served by recomputing the request's prefix without the connector, so any of these is a transport fault rather than a cache miss.
maxserve_cache_connector_loads_refused_loads_total	Counter	Cumulative loads the KV connector refused, from its host and disk tiers or from dKV. Each refused load is served by recomputing its blocks, so a steady rate is hit rate the tiers reported and could not deliver.
maxserve_cache_connector_offload_blocks_dropped_blocks_total	Counter	Cumulative offloaded KV blocks the connector's host pool had no room for. A pool starved by blocks pinned for in-flight transfers drops offloads before its hit rate falls.
maxserve_cache_connector_cordon_wait_time_milliseconds	Histogram	Time a prompt is held out of the batch on its KV connector, from the first time it is cordoned on a deferred connector lookup or an onload to its admission to a batch. Recorded once for each admitted prompt that was cordoned.
maxserve_cache_connector_cordon_passes	Histogram	Scheduler passes from a prompt's first cordon on a deferred connector lookup or an onload to its admission to a batch, recorded with maxserve_cache_connector_cordon_wait_time_milliseconds. A prompt admitted in the pass that cordoned it records 0.
maxserve_cache_connector_lookup_admission_stops_total	Counter	Prompt admission passes that stopped on a deferred connector lookup, counted per data-parallel replica, split by reason: budget when the pass had spent its share of lookup tokens, max_pending when the replica already had its bound of unanswered lookups out, and held when the next prompt arrived behind one still out on its lookup. The prompt waits at the head of its queue for a later pass.

Data parallel

MAX records these metrics only when data_parallel_degree > 1. The maxserve_ce_pool_wait_time_milliseconds metric also requires the data-parallel context-encoding balancer, which is off by default. To turn it on, set --dp-ce-balance-timeout-ms to 0 or higher.

Metric	Type	Description
maxserve_dp_active_token_occupancy_percent	Histogram	Per-batch data-parallel balance, measured as the mean/max of per-rank active-token load as a percentage. 100% means perfectly balanced ranks.
maxserve_dp_context_token_occupancy_percent	Histogram	Per-batch data-parallel balance, measured as the mean/max of per-rank context-token (KV/attention) load as a percentage. 100% means perfectly balanced ranks.
maxserve_dp_active_tokens_total	Counter	Cumulative active tokens scheduled across all data-parallel replicas, excluding padding. Divide by maxserve_dp_step_capacity_tokens_total over the same window for token-weighted DP occupancy.
maxserve_dp_step_capacity_tokens_total	Counter	Cumulative synchronized step capacity. For each batch, this is the data-parallel degree times the heaviest rank's active tokens.
maxserve_ce_pool_wait_time_milliseconds	Histogram	Time a CE request spends in the data-parallel balancer's unbound pool, from arrival until it is bound to a replica.

Vision and video

The following metrics track vision and video encoder activity and cache performance.

Metric	Type	Description
maxserve_vision_images_encoded_images_total	Counter	Cumulative images run through the vision encoder (cache misses).
maxserve_vision_images_cached_images_total	Counter	Cumulative images served from the vision encoder cache (cache hits).
maxserve_vision_patches_encoded_patches_total	Counter	Cumulative image patches fed to the vision encoder.
maxserve_vision_tokens_encoded_tokens_total	Counter	Cumulative merged vision tokens produced by the vision encoder.
maxserve_vision_cache_hit_rate_percent	Histogram	Per-batch vision encoder cache hit rate (0-100%).
maxserve_vision_preprocess_cache_hits_images_total	Counter	Cumulative images found already preprocessed, whose decode and preprocessing are both skipped.
maxserve_vision_preprocess_cache_misses_images_total	Counter	Cumulative images not already preprocessed, which the API server must decode and preprocess.
maxserve_vision_image_admission_decode_time_milliseconds	Histogram	Per-request wall-clock time of the admission image work: deciding which images are already preprocessed, then decoding and validating the rest.
maxserve_video_clips_encoded_clips_total	Counter	Cumulative video clips run through the video encoder (cache misses).
maxserve_video_tokens_encoded_tokens_total	Counter	Cumulative merged video tokens produced by the video encoder.
maxserve_video_encoding_time_milliseconds	Histogram	Per-batch video encoder wall-clock time.
maxserve_video_frames_per_clip_frames	Histogram	Sampled frame count per newly encoded video clip.

maxserve_vision_preprocess_cache_hits_images_total and its _misses counterpart are recorded from one of two vantages depending on the architecture, and the two are not the same measurement. Where the API server can ask the tokenizer at admission which images it has already preprocessed, the pair comes from that peek, before any media work runs. Every other architecture reports it from the tokenizer's own preprocess-cache lookup instead, published after tokenization. Both count images and neither carries a media_kind tag, so only the image cache feeds them. Because the windows differ, don't compare the rate across architectures.

Media ingestion

The following metrics track the work the API server does on client-supplied media before a request reaches the model: resolving each item to bytes, decoding images, and caching the preprocessed result. This window is inside maxserve_time_to_first_token_milliseconds.

Metric	Type	Description
maxserve_media_items_total	Counter	Cumulative media items the API server resolved into bytes, split by media_kind (image, video) and source (inline for a data URI, url for http(s), file for a local path). A rejected item is counted by maxserve_media_rejections_total instead.
maxserve_media_resolve_time_milliseconds	Histogram	Per-item time to turn one media reference into bytes: the http(s) download, the base64 decode of a data URI, or the local file read. Excludes the image decode. Split by source (inline, url, file). Items on one request are resolved concurrently, so these windows overlap and their sum isn't the request's elapsed resolve time.
maxserve_media_item_size_bytes	Histogram	Encoded size of one resolved media item as it arrived on the wire or in the request body. Not the decoded pixel buffer. Split by media_kind.
maxserve_media_image_size_pixels	Histogram	Pixel count (width * height) of one image as it arrived, read from the header before any resize, split by format.
maxserve_media_image_decodes_total	Counter	Cumulative images whose pixels the API server actually decoded, split by format. An image the preprocess cache let the server skip is sized but not counted here.
maxserve_media_image_decode_ms_total	Counter	Cumulative milliseconds of per-image codec decode, split by format. Excludes the per-architecture processor (resize, patchify, normalize), whose cost follows the pixel count rather than the container.
maxserve_media_rejections_total	Counter	Cumulative media items the API server refused, split by reason. Every value is an HTTP 400 to the client, so this counts client-visible failures rather than server faults.
maxserve_media_items_per_request_items	Histogram	Media items of one media_kind carried by a single request, counted before any media is downloaded and before the per-request count caps, so a request refused for carrying too many items is still sampled. Requests carrying none of that kind aren't sampled, so the distribution starts at one.
maxserve_media_preprocess_cache_evictions_total	Counter	Cumulative preprocessed-media entries the tokenizer dropped to stay inside its byte budget, split by media_kind. Budget pressure only, so an idle-timeout reclaim isn't counted.
maxserve_media_preprocess_cache_size_bytes	Gauge	Host bytes the tokenizer's preprocessed-media cache retains, split by media_kind. Sampled after tokenization, once per media request rather than on a timer.
maxserve_media_preprocess_cache_capacity_bytes	Gauge	The byte budget maxserve_media_preprocess_cache_size_bytes is evicted down to, split by media_kind. Zero means caching is disabled for that kind.

maxserve_media_image_decodes_total and maxserve_media_image_decode_ms_total are a counter pair rather than a histogram: their quotient is the mean decode cost for a format, and either reads on its own. Both carry the _total suffix Prometheus appends to a counter. The millisecond counter declares no unit, which is why its name ends in _ms_total rather than _milliseconds_total: with a declared unit, a backend that renders a counter as .count would spell it identically to the decode count beside it while it carried milliseconds.

maxserve_media_rejections_total carries a closed reason vocabulary, and reason is what makes the counter actionable: an SSRF block and a corrupt JPEG call for opposite responses. The values, by what they tell you:

• Unreachable or uncooperative source: dns_failure, fetch_status, fetch_timeout, fetch_transport, too_many_redirects, bad_redirect.
• Refused address or scheme: ssrf_blocked, bad_scheme, malformed_url, no_host, bad_file_uri.
• Over a configured limit: over_budget, decode_too_large, too_many_images, too_many_videos.
• Unusable payload: undecodable, bad_data_uri.

The two cache gauges export in bytes. Read maxserve_media_preprocess_cache_size_bytes against maxserve_media_preprocess_cache_capacity_bytes for occupancy, then use maxserve_media_preprocess_cache_evictions_total to explain a low hit rate on the preprocess cache: little eviction means the workload has no repeats, while steady eviction means the cache is thrashing at its budget. The two have opposite remedies.

Speculative decoding

The following metrics track draft-token acceptance rates for speculative decoding.

Metric	Type	Description
maxserve_spec_decode_acceptance_rate_per_position_percent	Histogram	Draft-token acceptance rate per position (0-100%).
maxserve_spec_decode_avg_acceptance_length_tokens	Histogram	Mean draft-token acceptance length per speculative decoding batch.

For more information, see speculative decoding.

Disaggregated inference

MAX records these metrics only in disaggregated inference deployments.

Metric	Type	Description
maxserve_di_api_pre_tokenize_milliseconds	Histogram	API-local time from HTTP accept to tokenize start (parse / route).
maxserve_di_api_submit_milliseconds	Histogram	API-local time from tokenize done to model-worker ZMQ put.
maxserve_di_mw_queue_wait_milliseconds	Histogram	Decode-worker time from first ZMQ pull of a request to admission-queue enqueue (drain gap).
maxserve_di_decode_admission_queue_wait_time_milliseconds	Histogram	Time a request spends in the decode node's admission queue before dispatch to prefill.
maxserve_di_decode_send_time_milliseconds	Histogram	Time to build and send a prefill request on the decode side (struct construction, serialization, ZMQ enqueue).
maxserve_di_dispatch_rtt_milliseconds	Histogram	Decode-clock round trip from sending a prefill request to receiving the prefill node's acknowledgment.
maxserve_di_prefill_span_milliseconds	Histogram	Decode-clock time between the prefill node's "arrived" and "ce_done" acknowledgments for a request. Covers prefill queue wait plus CE execution plus the return hop.
maxserve_di_reply_rtt_milliseconds	Histogram	Decode-clock time from the prefill node's "ce_done" acknowledgment to the actual prefill response arriving. Covers KV-transfer initiation, reply serialization, and network transit.
maxserve_di_prefill_queue_wait_time_milliseconds	Histogram	Prefill-local time a request spends enqueued before its first CE batch executes.
maxserve_di_decode_postprocess_time_milliseconds	Histogram	Decode-local time from receiving a prefill response to handing the result off to the response queue.
maxserve_di_early_sync_time_milliseconds	Histogram	Duration of the early-sync guard's blocking commit of the previous batch's structured-output FSM state, when it fired (see '_should_early_sync_prev_batch').
maxserve_di_ce_preempted_tg_iteration_count_total	Counter	Counter: incremented once per scheduling iteration in which a CE batch was admitted while TG requests were pending, so those TG requests got zero progress this iteration -- how often this preemption pattern occurs.
maxserve_di_ce_preempted_tg_pending_count	Histogram	Histogram: the number of TG requests pending at the moment of one such preemption (paired with 'maxserve.di.ce_preempted_tg_iteration_count', which counts how often it happens; this measures how large each occurrence is).
maxserve_di_handoff_to_first_token_time_milliseconds	Histogram	Wall-clock time from a prefill-to-decode handoff landing (its token discarded, re-queued as a one-token chunked-CE continuation) to decode's own re-forward actually sending token 1 to the API process. A TTFT contribution specific to the handoff path; the ordinary path's token already arrives with the 'PrefillResponse', so it has no equivalent wait.

Structured output and tool calls

The following metrics track schema-conformance errors and grammar-rejection rates for structured output and tool calls.

Metric	Type	Description
maxserve_tool_call_conformance_errors_total	Counter	Generated tool calls that failed the observability-only schema-conformance check, split by outcome label (invalid_json, unknown_tool, schema_mismatch).
maxserve_structured_output_grammar_rejections_total	Counter	Structured-output requests rejected at admission (HTTP 400) because the grammar backend couldn't compile the schema, split by kind label (tool_grammar, json_schema).
maxserve_structured_output_grammar_build_time_milliseconds	Histogram	Wall-clock time of an off-thread grammar-matcher build in AsyncGrammarGate, from the start of the backend's compile/create call to its completion. For a DI handoff, this build overlaps prefill's round trip when submitted at admission time; a build that outlasts that round trip can still gate the handoff's chunked-CE continuation.
maxserve_response_format_conformance_errors_total	Counter	response_format (json_schema/json_object) responses whose final content failed the observability-only schema-conformance check, split by outcome label (invalid_json, schema_mismatch).
maxserve_tool_call_responses_total	Counter	Count of chat-completion requests whose response contained at least one tool call, counted once per request however many calls it emitted. Against 'maxserve.tool_call.requests' this gives the share of declared tool inventories the model actually draws on; tools declared but never called cost prompt tokens on every request. Not a denominator for 'maxserve.tool_call.conformance_errors', which counts individual generated calls rather than requests.
maxserve_tool_call_tools_per_request_tools	Histogram	Distribution of tools declared per chat-completion request. Declared tools are rendered into the prompt, so inventory size explains a client's prompt length and grammar compile cost. Shares its population with 'maxserve.tool_call.requests'.
maxserve_tool_call_requests_total	Counter	Count of chat-completion requests that supplied a non-empty 'tools' array, split by the 'choice' tag (auto, required, named). Pairs with 'maxserve.tool_call.responses' to show how often a declared inventory is actually used. A tool_choice='none' request is not counted: it cannot produce a tool call.
maxserve_structured_output_requests_total	Counter	Count of chat-completion requests that carried a response_format, split by the 'kind' tag (json_object, json_schema); a 'text' format is the unconstrained default and is not counted. Counted before the schema is compiled, so a request later rejected by 'maxserve.structured_output.grammar_rejections' still counts.

Server

The following metrics track server startup and model loading.

Metric	Type	Description
maxserve_model_load_time_milliseconds	Histogram	Model-worker startup duration. Also split by component tag (build, compile, init, graph_capture, pinned_memory, spawn, total).
maxserve_pipeline_load_total	Counter	Pipelines loaded for each model.

Telemetry

By default, MAX collects anonymous usage metrics to help diagnose issues and improve performance. These metrics don't include personally identifiable information (PII).

To disable telemetry, set the MAX_SERVE_DISABLE_TELEMETRY environment variable:

docker run modular/max-nvidia-full:latest \
    -env MAX_SERVE_DISABLE_TELEMETRY=1 \
    ...

To send metrics to your own collector, set the standard OTEL_EXPORTER_OTLP_ENDPOINT environment variable. MAX exports traces only when OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is set. MAX_SERVE_OTLP_METRICS_ENDPOINT is a separate opt-in that additionally pushes an exponential-histogram shadow of every histogram; leave it unset unless you want that comparison.

To help the MAX team understand your deployment performance, you can add identity information with these environment variables:

• MAX_SERVE_DEPLOYMENT_ID: Your application name.
• MODULAR_USER_ID: Your company name.

For example:

docker run modular/max-nvidia-full:latest \
    -env MAX_SERVE_DEPLOYMENT_ID='Project name' \
    -env MODULAR_USER_ID='Example Inc.' \
    ...

Next steps

Learn more about deploying and optimizing MAX:

• Benchmark MAX on NVIDIA or AMD GPUs: Measure throughput, latency, and GPU utilization for your deployment.
• Deploy MAX on GPU with self-hosted endpoints: Set up a production deployment with the MAX container.
• Environment variables: Configure server settings, telemetry, and other runtime options.