A project showcasing how to leverage AI coding assistants (Cursor, Claude Code, etc.) for accelerated NVIDIA DeepStream SDK application development using a curated agentic skill and structured prompts.

Disclaimer: Code generated with AI coding assistants is intended as a development starting point. All generated code must undergo your full software development lifecycle (SDLC) — including code review, testing, and security validation — before production use.

• AI coding assistant that supports agentic skills (e.g., Cursor, Claude Code)

No GPU, SDK, or special hardware is required — the skill and example prompts work on any system.

The following are required on the target execution environment:

• NVIDIA DeepStream SDK — installed locally or available via NVIDIA NGC container
• Python 3.12+ with the pyservicemaker package
• NVIDIA GPU with driver version 595 or later
• CUDA 13.2 and TensorRT 10.16.0.72
• Supported OS: Ubuntu 24.04 (x86_64 or ARM64/Jetson)

The deepstream-import-vision-model and deepstream-eval-and-finetune skills run their toolchains inside nvcr.io/nvidia/deepstream:9.1-triton-multiarch; the host needs Docker, NVIDIA Container Toolkit, and a compatible NVIDIA driver. Their setup and preflight scripts create and verify the mounted Python environments and runtime tools.

For detailed environment setup, refer to the DeepStream SDK Developer Guide.

Some skills may document a specific DeepStream SDK version or container image that is not the latest DeepStream release version. Use the version documented by the skill unless you have a reason to test with a newer DeepStream image.

Skills that mention older DeepStream base images may still work out of the box with newer DeepStream versions. If you need to use a newer image, update the image tag and run the skill's setup, test, or benchmark steps to confirm the workflow.

This README sits under skills/ inside the DeepStream mono-repo. Layout of this subtree, plus the related top-level example_prompts/ directory:

deepstream/                                 # mono-repo root
├── skills/                                 # Agentic skills for guided DeepStream development
│   ├── README.md                           # This file
│   ├── deepstream-dev/                     # DeepStream development skill with condensed references
│   ├── deepstream-generate-pipeline/       # Interactive gst-launch pipeline builder (BM25 retrieval over 270+ pipelines)
│   ├── deepstream-profile-pipeline/        # Nsight Systems profiling & config derivation skill
│   ├── deepstream-import-vision-model/     # Autonomous vision-model onboarding & benchmarking pipeline skill
│   ├── deepstream-eval-and-finetune/       # DeepStream accuracy evaluation + AutoML fine-tuning skill
│   ├── deepstream-run-mv3dt/               # MV3DT reference-app operations skill
│   ├── deepstream-sop/                     # DeepStream SOP microservice skill (step-sequence compliance via GEBD + VLM)
│   ├── amc-setup-calibration-stack/        # AutoMagicCalib MS + UI launch skill
│   ├── amc-run-sample-calibration/         # AutoMagicCalib bundled-sample validation skill
│   ├── amc-run-video-calibration/          # AutoMagicCalib user-video calibration skill
│   ├── amc-run-rtsp-calibration/           # AutoMagicCalib RTSP-stream calibration skill
│   ├── rtvi-cv-customize-model/            # Swap CV detection model in RTVI-CV (VSS Alerts Blueprint)
│   ├── rtvi-vlm-customize-model/           # Swap VLM in the VSS Alerts Blueprint
│   └── rtvi-cv-scaffold-vss-service/       # Scaffold a custom RTVI CV microservice for VSS
└── example_prompts/                        # Pre-built prompts for code generation

This project provides the tooling and reference material needed to:

1. Supply domain-specific context to AI coding assistants through a curated agentic skill
2. Generate production-ready DeepStream code using well-structured prompts
3. Accelerate development of video analytics pipelines with AI assistance

An agentic skill is a structured knowledge package that an AI coding assistant can automatically discover and activate during code generation. It contains domain-specific rules, reference documentation, and guardrails that guide the AI agent to produce accurate, idiomatic code — without the developer needing to manually reference files in every conversation.

Each subdirectory under skills/ contains a DeepStream agentic skill that follows the standard SKILL.md convention supported by AI coding assistants such as Cursor, Claude Code, and others.

This project ships fourteen complementary skills:

Skip ahead to Skill: deepstream-import-vision-model for the model-onboarding workflow, or Skill: deepstream-run-mv3dt for the MV3DT reference-app workflow.

My DeepStream app dynamically adds and removes RTSP sources and latency rises below the configured streammux batch size. Diagnose it without writing application code.

This skill targets NVIDIA DeepStream SDK development using the Python pyservicemaker API. When activated, it instructs the AI agent to consult bundled reference documents before generating any code, significantly reducing inaccuracies and ensuring correct API usage.

Bundled reference topics:

Copy the deepstream-dev skill directory (including its references/ subdirectory) into the skills folder recognized by your AI coding assistant. You can install it at the user level (available across all projects) or at the workspace level (scoped to a single project).

# Example: Cursor user-level
mkdir -p ~/.cursor/skills/

# Example: Claude Code user-level
mkdir -p ~/.claude/skills/

# Example: Codex user-level
mkdir -p ~/.codex/skills/

# Example: workspace-level (replace .cursor with your tool's directory)
mkdir -p <workspace>/.cursor/skills/

# User-level (replace path with your tool's skills directory)
cp -r skills/deepstream-dev ~/.cursor/skills/

# Example: Claude Code user-level
cp -r skills/deepstream-dev ~/.claude/skills/

# Example: Codex user-level
cp -r skills/deepstream-dev ~/.codex/skills/

# Or workspace-level
cp -r skills/deepstream-dev <workspace>/.cursor/skills/

After copying, the directory structure should look like:

<skills-directory>/
└── deepstream-dev/
    ├── SKILL.md              # Skill definition with rules and quick references
    └── references/           # Condensed reference documents
        ├── best_practices.md
        ├── buffer_apis.md
        ├── gstreamer_plugins.md
        ├── kafka_messaging.md
        ├── media_extractor_advanced.md
        ├── nvinfer_config.md
        ├── rest_api_dynamic.md
        ├── service_maker_api.md
        ├── tracker_config.md
        ├── troubleshooting.md
        ├── use_cases_pipelines.md
        ├── utilities_config.md
        ├── metamux_config.md
        ├── docker_containers.md
        ├── streaming_sources.md
        └── nvds_msgapi_adapter.md

1. Open (or restart) your AI coding assistant.
2. Open the agent / chat panel.
3. Ask a DeepStream-related question, for example:

   Create a DeepStream pipeline that reads a video file and runs object detection using ResNet18.
   

4. The agent should automatically activate the deepstream-dev skill and consult its reference documents before generating code.

Tip: The skill is most effective in Agent mode. In agent mode, the AI assistant automatically selects and activates relevant skills based on the task context — no manual file referencing needed.

deepstream-generate-pipeline builds ready-to-run gst-launch-1.0 pipelines interactively. It collects pipeline requirements through a short questionnaire (input source, stream count, inference, tracker, sink, platform, extras), then assembles the pipeline using a standalone BM25 retrieval backend with structural metadata boosting over 270+ verified pipelines — pure Python stdlib, zero external dependencies. The result is validated (syntax, elements, properties, live parse) before it is presented.

Supported configurations:

Bundled scripts:

Bundled references: assembly-rules.md, output-format.md, requirement-extraction.md, security-and-limitations.md.

Install (same paths as deepstream-dev):

cp -r skills/deepstream-generate-pipeline ~/.cursor/skills/   # or ~/.claude/skills/ , ~/.codex/skills/

Try it — the skill activates from natural pipeline phrasing, no @-mention needed:

detect and track on 4 rtsp streams and display on jetson
give me a pipeline to infer on an image
build a pipeline

Tip: Works best in Agent mode. Run the bundled test suite with python3 -m unittest discover -s <skill-path>/tests -v.

deepstream-profile-pipeline replaces performance guesswork with measurement. When the user wants an efficient / fast / performant pipeline — or asks to benchmark, tune, or measure FPS — this skill profiles the pipeline with Nsight Systems (nsys profile + nsys stats, fully headless, no GUI), establishes two measured numbers (the inference plateau batch and the HW ceiling), and derives every other config from them. It then reports per-plugin NVTX timings for the end-to-end pipeline.

The skill is model- and pipeline-agnostic: it assumes only that the inference element is nvinfer or nvinferserver, and reads the user's actual config to discover model dims, target FPS, and source properties. It works for detection (with or without tracker), classification, segmentation, VLM, and embedding pipelines, with file / RTSP / USB-camera sources.

Bundled references:

Container requirement: Run from the nvcr.io/nvidia/deepstream:9.0-triton-multiarch dev image. The slimmer samples-multiarch variant strips the nsys NVTX injector and produces empty per-plugin traces — do not use it for profiling. Requires nsys (Nsight Systems 2024+) and nvidia-smi on PATH.

Install (same paths as deepstream-dev):

cp -r skills/deepstream-profile-pipeline ~/.cursor/skills/   # or ~/.claude/skills/ , ~/.codex/skills/

Try it:

build an efficient pipeline for this GPU
how many streams can this GPU handle at 30 FPS?
profile and tune my deepstream pipeline

Tip: Works best in Agent mode — it activates when a prompt carries efficiency/benchmarking intent.

deepstream-import-vision-model is an autonomous skill: instead of helping you write code, it executes a complete model bring-up pipeline and hands you back a benchmarked TensorRT engine plus a publication-ready PDF report.

Pipeline (runs unattended):

1. Model Acquire — parses a HuggingFace or NVIDIA NGC URL, downloads ONNX (or exports SafeTensors → ONNX via optimum-cli), extracts labels.
2. Engine Build — builds a dynamic TensorRT engine via trtexec, with iterative batch-size scaling and warm-cache reuse.
3. DeepStream Pipeline — generates a custom nvinfer bbox parser, builds the .so, runs single-stream KITTI validation, then a multi-stream sweep.
4. Report — produces 5 benchmark charts and a Markdown / HTML / PDF report under models/<model_name>/reports/.

Supported model scope: Object detection models only. Classification, segmentation, pose estimation, and other vision tasks are rejected up front (architecture detected from config.json — downloaded for HuggingFace models, extracted from the archive for NGC models).

The following detection architecture families are supported:

• Transformer-based detectors — query-based encoder-decoder designs (e.g., DETR and RT-DETR family, including TAO Transformer variants from NGC)
• One-stage grid-based detectors — single-pass, anchor-based or anchor-free designs that predict boxes directly from spatial feature grids (e.g., YOLO family)
• Open-vocabulary / zero-shot detectors — vision-language models that localize objects described by free-form text queries at inference time (e.g., GroundingDINO / OWL-ViT)

Models that fall outside these families are untested; custom bbox parsers may need manual adjustment for novel output tensor layouts.

Same install paths as deepstream-dev:

# Example: Cursor user-level
cp -r skills/deepstream-import-vision-model ~/.cursor/skills/

# Example: Claude Code user-level
cp -r skills/deepstream-import-vision-model ~/.claude/skills/

# Example: Codex user-level
cp -r skills/deepstream-import-vision-model ~/.codex/skills/

# Or workspace-level
cp -r skills/deepstream-import-vision-model <workspace>/.cursor/skills/

After copying:

<skills-directory>/
└── deepstream-import-vision-model/
    ├── SKILL.md            # Top-level skill definition + critical rules
    ├── references/         # 4 phase references (model-acquire, engine-build, pipeline-run, report-generation)
    └── scripts/            # Helpers: model/, engine/, deepstream/, report/

1. Open (or restart) your AI coding assistant on a workspace where you want bring-up artifacts to land (the skill writes to models/<model_name>/ relative to the project root).

2. Ask:

   Use deepstream-import-vision-model to onboard and benchmark this detection model
   end-to-end, and produce the PDF benchmark report:
   https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/rtdetr_2d_warehouse
   
   Use the default sample video at
   /opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4.
   

   In Cursor:

   @deepstream-import-vision-model onboard and benchmark this detection model end-to-end,
   and produce the PDF benchmark report:
   https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/rtdetr_2d_warehouse
   
   Use the default sample video at
   /opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4.
   

   For an interactive variant that prompts for inputs with defaults, see example_prompts/import_vision_model_detection_pipeline.md.

3. The agent should activate deepstream-import-vision-model, run the containerized setup and preflight checks (Docker, GPU, DeepStream 9.1, Python environment, TensorRT, and report tooling), and proceed through Steps 1–8 without further prompting.

models/<model_name>/
  model/          # ONNX file(s)
  parser/         # custom nvinfer bbox parser (.cpp, .so)
  config/         # nvinfer config, ds-app config, labels.txt
  benchmarks/     # TRT engines + trtexec / DS logs
  reports/        # benchmark_report.md / .html / .pdf + charts/
  samples/        # output .mp4, KITTI detections, test frames

The final PDF (reports/benchmark_report_<model_name>.pdf) being >500 KB is the skill's own success signal that charts were embedded correctly.

Tip: Like deepstream-dev, this skill works best in Agent mode. Manual @-mention is not required after install — the assistant picks it up from the URL pattern in your prompt.

deepstream-run-mv3dt operates the DeepStream Multi-View 3D Tracking reference app in src/apps/reference_apps/deepstream-tracker-3d-multi-view. It supports setup checks, shipped 4-camera and 12-camera sample runs, synchronized MP4 datasets, AutoMagicCalib handoff for missing calibration, display or headless execution, OSD/BEV outputs, and Kafka metadata inspection.

Same install paths as deepstream-dev:

# Example: Cursor user-level
cp -r skills/deepstream-run-mv3dt ~/.cursor/skills/

# Example: Claude Code user-level
cp -r skills/deepstream-run-mv3dt ~/.claude/skills/

# Example: Codex user-level
cp -r skills/deepstream-run-mv3dt ~/.codex/skills/

# Or workspace-level
cp -r skills/deepstream-run-mv3dt <workspace>/.cursor/skills/

Ask for an MV3DT workflow, for example:

Run the DeepStream MV3DT 4-camera sample headlessly and save output artifacts.

The agent should activate deepstream-run-mv3dt, resolve the MV3DT reference-app directory, run preflight checks, request approval before privileged Docker launch commands, and verify current-run artifacts rather than stale files.

Deploy MV3DT on the default sample dataset.

Run the MV3DT 12-camera sample headlessly with RTDETR and save videos.

Run MV3DT on synchronized MP4s under /path/to/my-dataset using PeopleNetTransformer.

My MV3DT videos under /path/to/my-dataset do not have calibration. Use AutoMagicCalib with resnet, then run MV3DT.

deepstream-sop is a domain-specific skill for building, deploying, evaluating, debugging, and measuring latency on the DeepStream SOP (Standard Operating Procedure) Inference Microservice — a GPU-accelerated FastAPI service that combines GEBD (Generic Event Boundary Detection, e.g. DDM) with VLM-based step classification (Cosmos Reason 1/2 via embedded vLLM) to verify operator step sequence on industrial video. It supports file, RTSP, and Basler GigE camera inputs, and emits results over SSE and/or Kafka.

The skill ships an 18-section progressive-disclosure reference layout: the agent loads only the sections relevant to the current task (generate, evaluate, latency-measurement, etc.), not the whole thing.

Bundled reference topics:

Generate the microservice scaffold:

Please follow @references/example_sop_prompt.md to generate a SOP microservice
in a folder named ds_sop_microservice.

Evaluate the microservice and fix issues until the test suite is green:

Please follow @references/eval_sop_prompt.md to evaluate the SOP microservice
in ds_sop_microservice. Fix any issues found.

Live-camera question — the skill activates without being named:

How do I send a chat completion request using a Basler GigE camera with serial 40748152?

Measure file-input latency:

Measure TTFC and C2C latency on /path/to/test_video.mp4 — follow the deepstream-sop skill.

Tip: Like the other skills, deepstream-sop works best in Agent mode — the assistant picks it up automatically when a prompt matches its trigger list (SOP / step-sequence / VLM-on-video / Basler camera / SOP-checker, etc.).

The AMC skills provide guided AutoMagicCalib setup and calibration workflows. When these skills mention the root README.md, assets/, compose/, projects/, or models/, they mean the resolved auto-magic-calib checkout. The setup skill finds an existing checkout or asks before cloning one.

These AMC skills are runtime-operation skills. Running setup or calibration requires a host with an NVIDIA GPU with NVENC, NVIDIA driver, Docker without sudo, NVIDIA Container Toolkit, and NGC image access. Optional VGGT refinement also requires HuggingFace access and the VGGT model. In restricted or no-GPU sandboxes, use the skills for planning and command preparation only.

Copy all AMC skill directories into your assistant's skill directory. Examples:

# Claude Code user-level
mkdir -p ~/.claude/skills
cp -r skills/amc-* ~/.claude/skills/

# Codex user-level
mkdir -p ~/.codex/skills
cp -r skills/amc-* ~/.codex/skills/

Restart or reopen the coding assistant after copying the skills.

launch auto calibration

calibrate sample dataset

calibrate these videos - /path/to/NV-Warehouse-4Cam/videos

calibrate these streams - rtsp://<host>:<port>/<path>/cam_00.mp4, rtsp://<host>:<port>/<path>/cam_01.mp4, rtsp://<host>:<port>/<path>/cam_02.mp4. Use resnet; I will upload alignment/layout/settings in the UI.

rtvi-cv-customize-model covers swapping the DeepStream CV detection model in the VSS Alerts Blueprint verification (2d_cv) mode. It walks through ONNX staging, custom bbox parser build and symbol alignment, nvinfer config, runtime TRT engine build via trtexec, and redeployment of perception-alerts. YOLO11 is the reference example; the same steps apply to any ONNX-format detector.

# Claude Code user-level
cp -r skills/rtvi-cv-customize-model ~/.claude/skills/

# Codex user-level
cp -r skills/rtvi-cv-customize-model ~/.codex/skills/

Replace the VSS Alerts Blueprint verification-mode detector with a custom ONNX model and redeploy perception-alerts.

How do I use a different detection model with the VSS Alerts Blueprint?

rtvi-vlm-customize-model covers swapping the VLM in the VSS Alerts Blueprint. It handles all three VLM consumers (rtvi-vlm, vlm-as-verifier, vss-agent), both deployment methods (OpenAI-compatible endpoint and in-container vLLM), and health checks.

# Claude Code user-level
cp -r skills/rtvi-vlm-customize-model ~/.claude/skills/

# Codex user-level
cp -r skills/rtvi-vlm-customize-model ~/.codex/skills/

Point the VSS Alerts Blueprint at a host-side NIM on http://host.docker.internal:30082 for all RTVI-VLM calls.

Run rtvi-vlm standalone with Qwen3-VL-8B-Instruct served inside the container.

rtvi-cv-scaffold-vss-service scaffolds a deployable custom RTVI CV microservice that plugs into VSS Search and Alerts profiles via Kafka mdx-raw. The shipped scaffold is a YOLO26 reference implementation and generates a runnable repo with a Dockerfile, DeepStream pipeline configs, compose definition, Python adapter, tests, and a smoke-test consumer.

# Claude Code user-level
cp -r skills/rtvi-cv-scaffold-vss-service ~/.claude/skills/

# Codex user-level
cp -r skills/rtvi-cv-scaffold-vss-service ~/.codex/skills/

Create a custom RTVI CV microservice that runs YOLO26 in DeepStream and publishes object metadata so VSS Search and Alerts can consume it.

Scaffold a custom perception microservice for VSS Alerts using my YOLO26 ONNX and wire it into mdx-raw.

The example_prompts/ directory contains pre-built prompts for generating DeepStream applications. Each prompt file provides a complete specification that an AI agent can follow to produce working code.

Getting started? Begin with video_infer_app.md for a minimal single-stream inference example, then progress to multi_stream_tracker.md for multi-stream and tracking capabilities.

Open the agent or chat panel in your AI coding assistant. Most tools provide a keyboard shortcut or sidebar icon for this.

Use your tool's file-referencing feature (e.g., @ mentions) to include the prompt file:

@example_prompts/rtvi_vlm_core_app.md

Or simply type @ and start typing the filename to search.

Option A: Direct execution

Reference the file in the chat and instruct the agent to follow it:

Follow the instructions in @example_prompts/rtvi_vlm_core_app.md to generate the application.

Option B: Incremental execution

For complex prompts, break them into smaller steps:

Based on @example_prompts/rtvi_vlm_core_app.md, first implement the vLLM backend module.

Then follow up with:

Now implement the frame selection logic as described in the prompt.

1. Review the generated code in the diff view.
2. Accept or reject individual changes.
3. Ask follow-up questions for refinements:

   Can you optimize the GPU memory usage in the generated stream_processor.py?
   

Here's a complete workflow for generating the RTVI VLM application:

1. Generate Core Application

@example_prompts/rtvi_vlm_core_app.md

Generate the complete application following these instructions.

2. Add FastAPI Microservice

@example_prompts/rtvi_vlm_openapi_spec.md

Create the FastAPI server with all endpoints shown in @rtvi_vlm_openapi_spec.png

1. Be specific — Include exact requirements, constraints, and expected outputs
2. Reference context — Use @ mentions to include relevant files and documents
3. Break down complex tasks — Divide large features into smaller, focused prompts
4. Include examples — Show expected input/output formats when applicable
5. Specify the deployment target — Mention whether the application targets dGPU (x86_64) or Jetson (ARM64), as pipeline elements and sink choices may differ

1. Review before accepting — Always inspect generated pipelines for correct element linking and property values
2. Test incrementally — Run the pipeline after each major change rather than building the entire application at once
3. Use the troubleshooting reference — If a pipeline fails, ask the agent to consult troubleshooting.md for known error patterns
4. Provide error output — When debugging, paste the full GStreamer or DeepStream error log into the chat for more accurate fixes