Official Implementation of EAGLE-1 (ICML'24), EAGLE-2 (EMNLP'24), and EAGLE-3 (NeurIPS'25).
-
Updated
Feb 20, 2026 - Python
#
speculative-decoding
Here are 485 public repositories matching this topic...
The fastest way to run Qwen 3.8 Flash Next, Qwen 3.8 27B and Ternary Bonsai 2 27B on a Mac: 125 tok/s in OpenCode on an M5 Max, and a 27B model on 16 GB Macs. Native MTP speculative decoding on Apple Silicon, exact at any temperature. OpenAI and Anthropic compatible local server.
macos metal mtp mlx inference-engine apple-silicon local-llm llm-inference local-ai qwen speculative-decoding openai-compatible claude-code qwen3-next anthropic-compatible native-mtp mtplx qwen3-8 qwen3-8-flash-next flash-next
-
Updated
Sep 26, 2026 - Python
⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡
retrieval chatbot rag habana large-language-model chatpdf llm-inference 4-bits speculative-decoding llm-cpu streamingllm intel-optimized-llamacpp neural-chat neural-chat-7b autoround gaudi3
-
Updated
Oct 8, 2024 - Python
Large-scale LLM inference engine
-
Updated
Sep 11, 2026 - Python
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
cuda quantization inference-optimization kv-cache vllm local-llm llm-inference qwen speculative-decoding qwen3 consumer-gpu rtx-3090
-
Updated
Sep 30, 2026 - Python
Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.
audio eagle quantization diffusion vlm llm qwen speculative-decoding llm-compression hunyuan deepseek fp4 dflash
-
Updated
Oct 1, 2026 - Python
A local inference engine for Apple silicon, built around the model.
-
Updated
Oct 1, 2026 - Python
Up to 4× faster LLM decoding on Apple Silicon, lossless. Native MLX port of DeepSeek's DSpark & z-lab's DFlash speculative decoding — Gemma-4, Qwen3.8, Muse-Glimmer, Nemotron, LFM2.5, Ornith-1.0, ternary Bonsai-27B.
macos metal pi codex mlx inference-engine apple-silicon local-llm llm-inference speculative-decoding claude-code dspark qwen3-8 dflash2
-
Updated
Sep 28, 2026 - Python
From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.
rust cuda pytorch triton quantization knowledge-distillation inference-engine jax kv-cache ml-systems llm mechanistic-interpretability fsdp flash-attention speculative-decoding paged-attention
-
Updated
Jun 5, 2026 - Python
Frontier-class open models on a free Kaggle TPU v5e-8: GLM-5.3-Flash 320B MoE (~64 tok/s, our own JAX engine) and Qwen3.8-27B bf16 (~130 tok/s), 262k context, prefix caching. Works with Claude Code, Codex, opencode and pi.
kaggle glm pallas tpu jax mixture-of-experts free-gpu openai-api long-context llm-serving vllm llm-inference qwen speculative-decoding coding-agents anthropic-api claude-code qwen3 glm-5-3-flash tpu-inference
-
Updated
Sep 15, 2026 - Python
[NeurIPS 2026 Spotlight] Fast, lossless LLM inference via dual-view diffusion decoding.
model-architecture efficient-inference large-language-models llm llm-inference speculative-decoding diffusion-language-models llm-efficiency
-
Updated
Aug 12, 2026 - Python
Fully uncensored, capability-enhanced abliteration of Qwen3.6-27B. NVFP4 + z-lab DFlash speculative decoding (n=12) on the unified ghcr.io/aeon-7/aeon-vllm-ultimate:latest container, tuned for long-context draft acceptance on DGX Spark. 6 HF variants (BF16/NVFP4/MTP/MTP-XS), docker-compose, and QuickStart.
quantization uncensored blackwell llm vllm qwen speculative-decoding abliteration qwen3 nvfp4 dgx-spark dflash
-
Updated
Sep 15, 2026 - Python
Code for "LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding", ACL 2024
-
Updated
Jul 20, 2026 - Python
scalable and robust tree-based speculative decoding algorithm
-
Updated
Jan 28, 2025 - Python
[COLM 2024] TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
-
Updated
Aug 31, 2024 - Python
A unified, torch-native training framework for MTP and block-parallel speculative decoding.
-
Updated
Jul 31, 2026 - Python
Qwen 3.8 27B ROCmFP4 on AMD Strix Halo (Ryzen AI Max+ 395). Up to 36 tok/s via MTP Speculation, TurboQuant & Mesa RADV Wave64.
vulkan mtp rocm llama-cpp qwen gguf speculative-decoding fp4 strix-halo amd-ryzen-ai radv turboquant
-
Updated
Sep 7, 2026 - Python
JetSpec: Breaking the Scaling Ceiling of Speculative Decoding with Causal Parallel Tree Drafting
-
Updated
Sep 21, 2026 - Python
Fastest measured Qwen3.8-27B config for DGX Spark (GB10): SGLang + NVFP4 + DFlash2, 72 tok/s greedy median, 1M context, deterministic boots. Flash-Next 176B on one box. One command, everything pinned, and a local cockpit to drive it.
-
Updated
Oct 2, 2026 - Python
Performance-tuned llama.cpp for AMD Strix Halo (gfx1151): FA + MoE-prefill fixes with a bundled current Mesa driver. Vulkan and HIP; portable dir, Docker, and distrobox.
amd vulkan rocm llama-cpp local-llm llm-inference flash-attention speculative-decoding strix-halo gfx1151 ryzen-ai-max radv
-
Updated
Sep 16, 2026 - Python
Add this topic to your repo
To associate your repository with the speculative-decoding topic, visit your repo's landing page and select "manage topics."