Qwen3.8-2.4T (1-bit) on 4x NVIDIA DGX Spark: native MTP self-speculative decode via llama.cpp RPC. ~8 prose / 11 structured tok/s. The how, the traps, the config.
-
Updated
Aug 13, 2026 - Shell
#
speculative-decoding
Here are 781 public repositories matching this topic...
Run Qwen3.8-27B locally on Apple Silicon with speculative decoding, driving the Qwen Code CLI. 40.6 tok/s on an M4 Pro, ~18 GB, no cloud.
-
Updated
Aug 20, 2026 - Shell
Event-driven benchmark of speculative decoding KV-cache overhead under memory pressure, comparing baseline, oracle, expected-reserve, and peak-reserve scheduler policies.
python benchmark research simulation latency scheduler inference systems gpu-memory throughput serving event-driven-simulation kv-cache llm speculative-decoding
-
Updated
Jul 22, 2026 - Python
Accelerating LLM inference on Jetson Orin Nano 8GB with llama.cpp, CUDA, and MTP speculative decoding
-
Updated
Aug 12, 2026 - Shell
长上下文推测解码的收益判据与增量回退调度（RAISE）参考实现 — Reference implementation of a break-even criterion and RAISE scheduling for long-context speculative decoding
transformers pytorch efficient-inference inference-optimization long-context llm-inference speculative-decoding longbench
-
Updated
Sep 22, 2026 - Python
a super fast llm response using small llm model to prefix large llm model
-
Updated
Jul 20, 2025 - Python
Suffix-tree speculative decoding for Hugging Face Transformers with cross-request caching and generation-pipeline integration.
-
Updated
Aug 7, 2026 - Python
vLLM serving stack for Gemma 4 31B on RTX PRO 6000 Blackwell, with FP8 KV cache, MTP speculative decoding, and an async FastAPI logging proxy in front.
-
Updated
Jun 22, 2026 - Shell
LLM inference optimization experiments: speculative decoding, continuous batching, KV-cache, quantization
python benchmarking transformers inference pytorch quantization llm vllm speculative-decoding continuous-batching
-
Updated
May 25, 2026 - Python
Simulates speculative decoding to find the optimal speculation length K across 576 configurations (3 draft models x 8 K values x 6 acceptance rates x 4 cost ratios). Key findings: 6.06x max speedup, breakeven at cost_ratio=0.25, optimal K grows from 1-3 at 50% acceptance to 7-15 at 95% acceptance.
simulation gpu optimization inference transformer throughput cpp20 llm speculative-decoding draft-model
-
Updated
Jul 10, 2026 - Python
Mini LLM inference server: paged KV cache, continuous batching, prefix caching, speculative decoding, OpenAI-compatible API
python inference-server machine-learning-systems kv-cache llm-inference speculative-decoding paged-attention continuous-batching
-
Updated
Aug 13, 2026 - Python
Real implementation of speculative decoding showing that speedup is strongly prompt- and target-dependent: GPT-2 → GPT-2-medium achieves mean best speedup 1.013×, while GPT-2 → GPT-2-large reaches 1.253× and up to 1.846× on high-agreement prompts.
benchmarking transformers decoding pytorch gpt2 kv-cache llm-inference speculative-decoding gpu-profiling gpt2-large
-
Updated
Jul 13, 2026 - Python
Agent-Level Speculative Orchestration & Formal Dual-Engine Code Generation (Rust, FastMCP, DeepSeek)
python rust code-generation formal-verification speculative-decoding deepseek mcp-server fastmcp claude-code agentic-coding cursor-mcp deepseek-v4 llm-as-a-verifier dspark cegar dspark-cli
-
Updated
Sep 1, 2026 - Rust
Proof-carrying, high-performance LLM inference in Rust on fe2o3
-
Updated
Oct 7, 2026 - JavaScript
Controlled EAGLE-3 draft adaptation for frozen Qwen3-8B: Rust latency, replication, scaling controls, and serving trade-offs.
-
Updated
Sep 20, 2026 - Python
Split-KV for MTP / Speculative Decoding verify (seqlen_q=2) in FlashAttention-2 + W8A8 kernel tuning on Ada (5.8x attention speedup at 105K context)
-
Updated
Sep 19, 2026 - Python
Inference Systems Lab: 1,536 controlled vLLM requests on one RTX 4090, cache/concurrency comparisons, raw traces and bounded streaming benchmark tools.
-
Updated
Sep 25, 2026 - Jupyter Notebook
Benchmark open-weight LLMs on vLLM in Docker under one fixed protocol: decode at 0/4k/16k, cold TTFT, 4-way concurrency, power and VRAM, three runs with spread. Standard-library Python, charts and a results explorer.
python docker benchmark gpu inference quantization llm vllm local-llm speculative-decoding rtx-4090 llm-benchmark
-
Updated
Sep 26, 2026 - Python
Add this topic to your repo
To associate your repository with the speculative-decoding topic, visit your repo's landing page and select "manage topics."