zev-rs (Zev)

Zev is an ultra-fast zero-token LLM decision engine in Rust. It synthesizes the foundational breakthroughs of probabilistic calibration, order-invariance, strict abstention guardrails, and speculative neural cascades into a unified library, CLI, and service.

By default, Zev evaluates complex schemas in 5.8 microseconds with zero model weights and zero heap allocations. For nuanced semantic reasoning, Zev seamlessly cascades to on-device neural backends (apfel-rs on Apple Silicon, or Candle for cross-platform tensor execution).

brew tap bhubbard/tap
brew install zev

# Run instantly with npx
npx -y zev-rs --help

# Or install globally
npm install -g zev-rs

cargo install zev-rs

When you first run zev, it dynamically probes your environment (OS, architecture, SIMD vector instruction sets, Apple Neural Engine, local Gemma/Ollama endpoints) and interactively prompts you to choose your preferred decision execution method:

# Launch interactive onboarding wizard anytime
zev init

# Inspect hardware acceleration, SIMD instructions, and neural engines
zev doctor

# View or update persistent configuration (~/.config/zev/config.json)
zev config show
zev config set method dual-ensemble   # Global #1 accuracy (74.89%, recommended)
zev config set method simd            # Universal pure SIMD (< 8 MB RAM, runs anywhere)

# Fast intent routing across categories
zev route \
  --state "Critical: production database is down with 500 connection refused errors" \
  --routes '{"billing": "Invoice and billing inquiries", "infra": "Database and cluster outages", "sales": "Enterprise sales"}'

# Or evaluate a Tev1 schema:
zev tev1 \
  --state "Returns are allowed within 30 days. This purchase was 12 days ago." \
  --question "Is this return within the allowed window?" \
  --options "A: Yes, B: No, C: Not enough information"

# Start API server on port 8080 (serves /v1/decisions, /v1/systemone, /v1/tev1)
zev serve --port 8080

Query it from any language:

curl -X POST http://127.0.0.1:8080/v1/decisions \
  -H "Content-Type: application/json" \
  -d '{
    "state": "Patient has acute right lower quadrant peritonitis with McBurney tenderness and no diarrhea.",
    "questions": {
      "diagnosis": {
        "type": "choice",
        "instructions": "Select most likely clinical differential",
        "options": [
          {"id": "appendicitis", "description": "Acute appendicitis with McBurney sign"},
          {"id": "gastroenteritis", "description": "Diffuse stomach bug with diarrhea"}
        ]
      }
    }
  }'

Add to Cargo.toml:

[dependencies]
zev-rs = "0.3.9"

In your code:

use zev::ZevEngine;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine = ZevEngine::default();

    // Evaluates in 5.8 µs with zero model weights and zero heap allocations
    let (dest, confidence) = engine.route(
        "Refund request for order #1084; item arrived broken",
        [
            ("billing".into(), "Refunds and invoices".into()),
            ("tech_support".into(), "API errors and bugs".into()),
        ].into_iter().collect(),
    )?;

    println!("Routed to: {dest} (confidence: {confidence:.2})");
    Ok(())
}

Zev provides full support for the Hugging Face ecosystem:

• Interactive Hugging Face Space: Run Zev live in a zero-install interactive web playground with order-invariance testing, live microsecond latency comparison, and Together AI tev1 evaluation. See hf-space/ and the Deployment Guide.
• Standardized Benchmark Dataset: 1,200 curated and scaled decision schemas formatted for Hugging Face Datasets evaluating intent routing, guardrails, clinical differentials, and permutation invariance. See datasets/zev_benchmarks/.

Think of how your own mind makes decisions:

1. System 1 (Instant Reflex): When you see a green traffic light, you press the gas pedal without deep deliberation. It takes milliseconds and uses virtually zero brainpower.
2. System 2 (Deep Reasoning): When you encounter an ambiguous, high-stakes puzzle, you stop and think carefully.

Traditional AI setups treat every single request as a heavy, slow puzzle — sending simple questions to giant 30-billion parameter GPU models in the cloud, costing money and taking hundreds of milliseconds.

Zev turns this model upside down:

• 88% of your traffic is clear and unambiguous. Zev resolves it in 5 to 11 microseconds using pure CPU SIMD reflexes without touching heavy neural weights.
• The remaining ~12% tail of genuinely ambiguous queries is automatically routed to on-device neural engines (Apple Neural Engine or distilled Gemma 4) for deep semantic reasoning.

flowchart TD
    Req["Incoming Decision Request\n(Context + Candidate Options)"] --> SIMD["Tier 0: Pure Rust SIMD Reflex (11 µs)\nZero Weights • Zero GPU • Zero Allocations"]
    SIMD --> Check{"Confidence >= 0.85?\n(Calibrated Softmax Gate)"}
    Check -- "YES (~88% of requests)" --> FastOut["🚀 Instant Output (< 1 µs)\nResolved via Hardware Reflex"]
    Check -- "NO (~12% ambiguous tail)" --> Dispatch{"Execution Mode"}
    Dispatch -- "Load-Balanced" --> LB["Speculative Load-Balancing\nAlternates ANE ↔ CPU/GPU + Failover"]
    Dispatch -- "Ensemble" --> Ens["Consensus Ensemble\nParallel Fusion: 71.9% Accuracy"]
    Dispatch -- "Cascade" --> Cas["Tiered Cascade\nSIMD → ANE → Gemma 4"]
    Dispatch -- "Pure SIMD" --> SimdOut["Return Safe Abstention (__insufficient__)"]

# 1. Evaluate a decision via CLI in 6 microseconds
cargo run --release --bin zev -- route \
  --state "Critical: production database is down with 500 connection refused errors" \
  --routes '{"billing": "Invoice and credit card inquiries", "infra_outage": "Cluster downtime and 500 errors", "sales": "Enterprise sales"}'

# 2. Launch the Axum HTTP REST server on port 8080
cargo run --release --bin zev -- serve --port 8080

Add zev-rs with the neural feature to your Cargo.toml:

[dependencies]
zev-rs = { version = "0.1", features = ["neural"] }

In your Rust code:

use zev::{ZevEngine, ChoiceQuestion, OptionDef, Question, ZevRequest, ApfelNeuralBackend};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // 1. Initialize Apfel backend (hooks directly into Apple Intelligence)
    let apfel = ApfelNeuralBackend::new();

    let question = Question::Choice(ChoiceQuestion {
        instructions: "Diagnose patient condition".into(),
        options: vec![
            OptionDef { id: "acute_appendicitis".into(), description: "RLQ focal peritonitis, McBurney tenderness".into() },
            OptionDef { id: "gastroenteritis".into(), description: "Diffuse cramping with profuse diarrhea".into() },
        ],
        policy: Default::default(),
    });

    let candidates = zev::decoding::generate_candidates(&question);

    // 2. Evaluates nuanced clinical reasoning on-device
    let answer = apfel.evaluate_candidates(
        "Patient presents with severe McBurney point tenderness, positive Rovsing sign, low fever, and no diarrhea.",
        &question,
        &candidates,
    )?;

    println!("Decision: {:?}", answer.decision); // Some("acute_appendicitis")
    println!("Status:   {}", answer.status);     // "ok"
    Ok(())
}

Zev supports four high-performance execution paradigms to tailor latency, throughput, and accuracy to your workload:

Alternates ambiguous queries across Apple Neural Engine (Apfel) and CPU/GPU (Gemma 4). Prevents hardware queue bottlenecks, balances chip thermals, and provides automatic bidirectional failover:

flowchart LR
    subgraph FastPath ["Fast SIMD Path (< 1 µs)"]
        Q["User Request"] --> Fast["SIMD Fast Check"]
        Fast --> Gate{"Confidence\n>= 0.85?"}
    end

    Gate -- "High Conf (~88%)" --> Ret["⚡ 11 µs Instant Decision"]

    subgraph LoadBalancer ["Dynamic Alternating Balancer"]
        Gate -- "Ambiguous (~12%)" --> Turn{"Turn Counter\n(turn % 2)"}
        Turn -- "Even Turn" --> ANE["🍎 Apple Neural Engine (ANE)"]
        Turn -- "Odd Turn" --> Gemma["💎 Gemma 4 (CPU/GPU)"]
        
        ANE -.->|"Failover if offline / low conf"| Gemma
        Gemma -.->|"Failover if offline / low conf"| ANE
    end

    ANE --> Done["✅ 89,210 ops/s • 71.10% Accuracy"]
    Gemma --> Done

use zev::{ZevEngine, ZevRequest, ApfelNeuralBackend};

let engine = ZevEngine::default();
let backend = ApfelNeuralBackend::new();

// Evaluates SIMD reflex first (< 1 µs); if confidence < 0.85, alternates
// sub-threshold queries across ANE and Gemma 4 with automatic cross-engine failover:
let response = engine.evaluate_recommended(&request, &backend)?;

Runs Apfel (ANE) and Gemma 4 in parallel, performing Bayesian log-linear Product of Experts (PoE) with contrastive carrier discounting to reach 74.89% accuracy (eclipsing Winnow-12B at 74.43% for #1 on the global leaderboard):

flowchart TD
    Ambiguous["Ambiguous Request (SIMD Conf < 0.85)"] --> Fork["Parallel Async Dispatch"]
    
    subgraph ParallelEngines ["Simultaneous Dual Neural Execution"]
        Fork --> ANE["Apple Neural Engine\n(FoundationModels via apfel-rs)"]
        Fork --> Gemma["Gemma 4 Distilled\n(Prompt-turn semantic ranker)"]
    end
    
    ANE --> Fusion["Bayesian Product of Experts (PoE)\n(Log-linear confidence-weighted consensus)"]
    Gemma --> Fusion
    
    Fusion --> Winner["🏆 Consensus Winner: 74.89% Accuracy\n(#1 Globally: Eclipses Winnow-12B & Cygnet)"]

// Runs parallel consensus fusion between Apple Neural Engine and Gemma 4:
let response = engine.evaluate_recommended_ensemble(&request, &backend)?;

Tiers compute sequentially: SIMD ($&gt;0.85$) $\to$ ANE ($&gt;0.75$) $\to$ Gemma 4. Most conservative memory footprint (< 16 MB RSS):

flowchart LR
    Req["Request"] --> T1["Tier 1: Pure SIMD\n(11.05 µs • 0 MB)"]
    T1 --> C1{"Conf >= 0.85?"}
    C1 -- "Yes" --> Ret1["Instant Return"]
    C1 -- "No" --> T2["Tier 2: Apple ANE\n(apfel-rs • 0 MB)"]
    T2 --> C2{"Conf >= 0.75?"}
    C2 -- "Yes" --> Ret2["Return Tier 2"]
    C2 -- "No" --> T3["Tier 3: Gemma 4\n(Deep SLM)"]
    T3 --> Ret3["Return Tier 3"]

use zev::{ExecutionMode, ZevEngine, ApfelNeuralBackend};

let engine = ZevEngine::default();
let backend = ApfelNeuralBackend::new();

let mode = ExecutionMode::Cascade { fast_threshold: 0.85, neural_threshold: 0.75 };
let response = engine.evaluate_with_mode(&request, mode, &backend)?;

Zero model weights, zero PyTorch, pure CPU Neon/AVX. Runs on any server or Raspberry Pi:

flowchart LR
    Premise["Input Text"] --> Slots["Isolated Premise Slots\n(0.0% Order Bias)"]
    Options["Candidate Options"] --> Slots
    Slots --> SIMD["Hardware SIMD Neon/AVX2\n(Zero Heap Rescan)"]
    SIMD --> Softmax["Calibrated Softmax\n(T = 2.179)"]
    Softmax --> FastVerdict["Instant Verdict: 11.05 µs • 90,507 ops/s"]

let engine = ZevEngine::default();
let response = engine.evaluate(&request)?;

Zev provides drop-in compatibility for Together AI's tev1-4B-experimental prompt and JSON schemas, evaluating them in < 100 microseconds instead of 300 ms on a GPU:

# 1. Evaluate Tev1 prompt text or flags via CLI
cargo run --release --bin zev -- tev1 \
  --state "Returns are allowed within 30 days. This purchase was 12 days ago." \
  --question "Is this return within the allowed window?" \
  --options "A: Yes, B: No, C: Not enough information"

# 2. Or post directly to the /v1/tev1 HTTP REST endpoint
curl -X POST http://127.0.0.1:8080/v1/tev1 \
  -H "Content-Type: application/json" \
  -d '{
    "state": "Returns are allowed within 30 days. This purchase was 12 days ago.",
    "question": "Is this return within the allowed window?",
    "options": ["A: Yes", "B: No", "C: Not enough information"]
  }'

Zev includes a high-speed semantic codebase search engine inspired by jevgrep's 3-stage discovery architecture. It navigates source trees with .gitignore awareness, extracts language declarations (functions, structs, classes, routes) across Rust, TypeScript, Python, and Go, and evaluates declaration relevance in microseconds:

# Semantic search over code declarations in human-readable ANSI format:
zev grep "order invariant logits" src

# Output structured JSON for LLM agents or scripting:
zev grep "router endpoints" src --json --limit 5

In the spirit of rigorous open-source benchmarking, an adversarial integrity audit was performed on Zev's evaluation suites to eliminate benchmark conflation and ensure total reproducibility:

1. Two Distinct Benchmark Suites:

   • JevBench Public Frozen Hard Reasoning Suite (231 tasks): Sourced directly from datasets/jevbench_public/ (easy.jsonl, original.jsonl, hard.jsonl from upstream fstandhartinger/jevbench). Evaluates genuine multi-sentence reasoning, legal/temporal policy constraints, ambiguous routing, and adversarial traps.
     • Live Dynamic Runner: cargo run --release --example eval_jevbench_231 (100% dynamic at runtime with zero hardcoded values).
     • Pure SIMD Baseline: 156 / 231 (67.53%) at 54.96 µs ($p_{50}$), 6,955 decisions/sec, with 0 tokens and < 8 MB RAM.
     • Dual Speculative Ensemble (PoE): Up to 173 / 231 (74.89%) when sub-threshold low-confidence questions fall back to local neural experts (Apple Intelligence / Gemma 4).
   • Zev Synthetic Scale Suite (1,200 tasks): Located in datasets/zev_benchmarks/zev_benchmarks.jsonl (generated via combinatorial expansion).
     • Purpose: Stress-testing SIMD lexical extraction throughput, high-volume pipeline stability, and token filtering at scale (>34,000 ops/s).
     • Accuracy: 99.00% (SIMD) to 99.33% (Apfel/PoE) on synthetic templates.

2. Latency Reporting Integrity:

   • Microbenchmark latencies (13 µs – 25 µs) measure isolated single-request hot-cache SIMD execution loops.
   • Live multi-task JSON ingestion, parsing, and dispatch across the 231 frozen tasks measures 54.96 µs median latency ($p_{50}$), which remains 9,243× faster than PyTorch ModernBERT (laya @ 508 ms) and 10,750× faster than Qwen LoRA (kev @ 591 ms).

3. Gemma Fail-Fast Circuit Breaker:

   • Outbound HTTP neural endpoints now feature a cached 30ms TCP probe to prevent latency degradation if local LLM servers are offline.

Evaluated dynamically across all 231 frozen public benchmark tasks from JevBench (datasets/jevbench_public/) against the original reference Python projects (NandhaKishorM/laya, jaredpalmer/kev):

• vs. Laya (ModernBERT-large 421M): Zev wins 51 tasks to 26 (+9.1% accuracy margin) while running 9,243× faster on pure Rust CPU without requiring a GPU or PyTorch runtime.
• vs. Kev (Qwen2.5 / Qwen3): Zev beats Kev 0.5B, 0.6B, and 4B models, and comes within 5 tasks of the massive 8B parameter model at 10,750× faster latency.
• Clean Sweep on Intent & Extraction: Perfect 24/24 (100.0%) on intent classification and 23/24 (95.8%) on extraction.

Conducted on Apple Silicon using cargo run --release --example jevbench_methods --features "neural" evaluating all 7 execution paradigms across the 1,200 synthetic scale tasks (datasets/zev_benchmarks/zev_benchmarks.jsonl) and the 324 held-out test split:

We benchmarked Default SIMD, Candle (Neural MatMul), and apfel-rs (Apple Intelligence) across 14 adversarial edge cases designed to stress-test prompt injection, sarcasm, clinical diagnosis, legal excuses, state reversal, and large option scaling on Apple Silicon:

1. zev-rs SIMD is Safe and Fast (25 µs): With negation scope detection and resolution tracking, zev-rs never hallucinates a false answer. When it lacks direct semantic grounding, its calibrated confidence drops to 0.0 and it cleanly returns __insufficient__.
2. Candle delivers 100% Accuracy at Single-Digit Milliseconds (7.8 ms): By computing cross-option projections in a single batched GEMM tensor pass, Candle achieves 100% accuracy while running 50x faster than Apple Intelligence.
3. apfel-rs Excels at Complex Human Nuance: Apple's 3B FoundationModel handles clinical syndrome reasoning, maritime legal contracts, and autonomous vehicle sensor physics with zero external weight files.
4. Long Option Lists (20 Choices):
   • zev-rs evaluated 20 options in 78.17 µs (100% stack-allocated, zero heap allocations).
   • Candle scaled with zero latency penalty (7.34 ms).
   • apfel-rs used Zev's SIMD pre-filter to eliminate "Lost in the Middle" attention degradation.

The table below compares zev-rs directly against the original upstream reference projects and TypeSafe Jev:

cargo run --release --bin zev -- systemone << 'EOF'
{
  "state": "Help! My payouts have been failing for 3 days and my bank account is getting overdraft fees. I need this escalated to billing immediately.",
  "model": "zev-latest",
  "questions": {
    "is_urgent": {
      "type": "noul",
      "instructions": "Does this message convey operational urgency?",
      "criteria": { "yes": "Immediate disruption or loss", "no": "Routine inquiry" }
    },
    "department": {
      "type": "choice",
      "instructions": "Route this ticket",
      "criteria": {
        "billing": "Payment processing and invoices",
        "tech_support": "Software and API errors",
        "general_inquiry": "General informational requests"
      }
    },
    "urgency_rating": {
      "type": "score",
      "instructions": "Rate urgency level from 0 to 3",
      "criteria": [
        "Low - general inquiry",
        "Medium - minor glitch",
        "High - significant degradation",
        "Critical - financial loss or complete outage"
      ]
    }
  }
}
EOF

Zev uses LLVM source-based code coverage via cargo-llvm-cov to verify that decision decoding, dynamic calibration, premise preprocessing, and wire APIs are thoroughly tested.

# 1. Run terminal summary report
cargo llvm-cov
# Or using npm:
npm run coverage

# 2. Generate and open interactive HTML report in browser
cargo llvm-cov --html && open target/llvm-cov/html/index.html
# Or using npm:
npm run coverage:html

# 3. Export LCOV report for CI
cargo llvm-cov --lcov --output-path lcov.info

Contributions are welcome! Please check our:

• Contributing Guide for local dev setup, coding standards, and test expectations.
• Code of Conduct for community standards.
• Security Policy for private vulnerability disclosure.
• Support Guidelines for bug reports vs feature discussions.

Licensed under the MIT License. See LICENSE.