Skip to content

Commit 31d7ca5

Browse files
pimlockpimlock
authored
refactor(inference): simplify routing — introduce inference.local, remove implicit catch-all (NVIDIA#146)
Remove multi-route CRUD system and replace with single managed cluster route (inference.local). Key changes: - Remove inference route CRUD RPCs and CLI commands - Remove InspectForInference OPA action; policy is binary allow/deny - Introduce AuthHeader enum and InferenceProviderProfile in navigator-core - Router is now provider-agnostic: auth style carried on ResolvedRoute - Replace InferenceRouteSpec with ClusterInferenceConfig (2 fields vs 8) - Rename proto: routing_hint->name, SandboxResolvedRoute->ResolvedRoute, GetSandboxInferenceBundle->GetInferenceBundle, drop sandbox_id param - Rename RouteConfig.route -> RouteConfig.name; use inference.local - Add 'nemoclaw cluster inference update' for partial config changes - Delete stale navigator.inference.v1.rs checked-in proto file - Update architecture docs, agent skills, and CLI reference Closes NVIDIA#133
1 parent 91c7f84 commit 31d7ca5

51 files changed

Lines changed: 1788 additions & 3395 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/skills/generate-sandbox-policy/SKILL.md‎

Lines changed: 3 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -365,7 +365,7 @@ The policy needs to go somewhere. Determine which mode applies:
365365

366366
1. **Read the existing file** to understand current state:
367367
- What policies already exist under `network_policies`
368-
- What the `filesystem_policy`, `landlock`, `process`, and `inference` sections look like
368+
- What the `filesystem_policy`, `landlock`, and `process` sections look like
369369
- Whether the file uses compact (`{ host: ..., port: ... }`) or expanded YAML style
370370

371371
2. **Check for conflicts**:
@@ -377,7 +377,7 @@ The policy needs to go somewhere. Determine which mode applies:
377377
- **Modifying an existing policy**: Edit the specific policy in place — add/remove endpoints, change access presets, update rules, add binaries, etc.
378378
- **Removing a policy**: Delete the policy block if the user asks.
379379

380-
4. **Preserve everything else**: Do not modify `filesystem_policy`, `landlock`, `process`, `inference`, or other policies unless the user explicitly asks.
380+
4. **Preserve everything else**: Do not modify `filesystem_policy`, `landlock`, `process`, or other policies unless the user explicitly asks.
381381

382382
### Mode B: Create a New Policy File
383383

@@ -410,13 +410,9 @@ process:
410410

411411
network_policies:
412412
# <generated policies go here>
413-
414-
inference:
415-
allowed_routes:
416-
- local
417413
```
418414

419-
The `filesystem_policy`, `landlock`, `process`, and `inference` sections above are sensible defaults. Tell the user these are defaults and may need adjustment for their environment. The generated `network_policies` block is the primary output.
415+
The `filesystem_policy`, `landlock`, and `process` sections above are sensible defaults. Tell the user these are defaults and may need adjustment for their environment. Cluster inference is configured separately through `nemoclaw cluster inference set/get`. The generated `network_policies` block is the primary output.
420416

421417
If the user provides a file path, write to it. Otherwise, suggest `deploy/docker/sandbox/dev-sandbox-policy.yaml` for local development or ask where to place it.
422418

‎.agents/skills/generate-sandbox-policy/examples.md‎

Lines changed: 2 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -754,7 +754,7 @@ An exact IP is treated as `/32` — only that specific address is permitted.
754754
- { path: /usr/bin/curl }
755755
```
756756

757-
The agent uses `StrReplace` to insert after the last existing policy in the `network_policies` block. All other sections (`filesystem_policy`, `landlock`, `process`, `inference`) are untouched.
757+
The agent uses `StrReplace` to insert after the last existing policy in the `network_policies` block. All other sections (`filesystem_policy`, `landlock`, `process`) are untouched.
758758

759759
---
760760

@@ -866,13 +866,9 @@ network_policies:
866866
access: full
867867
binaries:
868868
- { path: /usr/local/bin/claude }
869-
870-
inference:
871-
allowed_routes:
872-
- local
873869
```
874870

875-
The agent notes that `filesystem_policy`, `landlock`, `process`, and `inference` are sensible defaults that may need adjustment.
871+
The agent notes that `filesystem_policy`, `landlock`, and `process` are sensible defaults that may need adjustment, and that cluster inference is configured separately via `nemoclaw cluster inference set/get` rather than an `inference` policy block.
876872

877873
---
878874

‎.agents/skills/nemoclaw-cli/SKILL.md‎

Lines changed: 20 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -436,46 +436,39 @@ nemoclaw sandbox delete work-session
436436

437437
---
438438

439-
## Workflow 7: Inference Routing
439+
## Workflow 7: Cluster Inference
440440

441-
Configure inference routes so sandboxes can access LLM endpoints.
441+
Configure the cluster's managed inference route for `inference.local`.
442442

443-
### Create an inference route
443+
### Set cluster inference
444+
445+
First ensure the provider record exists:
444446

445447
```bash
446-
nemoclaw inference create \
447-
--routing-hint local \
448-
--base-url https://my-llm.example.com \
449-
--model-id my-model-v1 \
450-
--api-key sk-abc123
448+
nemoclaw provider list
451449
```
452450

453-
If `--protocol` is omitted, the CLI auto-detects by probing the endpoint.
454-
455-
### List and manage routes
451+
Then point cluster inference at that provider and model:
456452

457453
```bash
458-
nemoclaw inference list
459-
nemoclaw inference update my-route --routing-hint local --base-url https://new-url.example.com --model-id my-model-v2
460-
nemoclaw inference delete my-route
454+
nemoclaw cluster inference set \
455+
--provider nvidia \
456+
--model nvidia/nemotron-3-nano-30b-a3b
461457
```
462458

463-
### Connect sandbox to inference
459+
This updates the cluster-managed `inference.local` route. There is no per-route create/list/update/delete workflow for sandbox inference.
464460

465-
Ensure the sandbox policy allows the routing hint:
461+
### Inspect current inference config
466462

467-
```yaml
468-
# In the policy YAML
469-
inference:
470-
allowed_routes:
471-
- local
463+
```bash
464+
nemoclaw cluster inference get
472465
```
473466

474-
Then create the sandbox with the policy:
467+
### How sandboxes use it
475468

476-
```bash
477-
nemoclaw sandbox create --policy ./policy-with-inference.yaml -- claude
478-
```
469+
- Agents send HTTPS requests to `inference.local`.
470+
- The sandbox intercepts those requests locally and routes them through the cluster inference config.
471+
- Sandbox policy is separate from cluster inference configuration.
479472

480473
---
481474

@@ -553,7 +546,8 @@ $ nemoclaw sandbox sync --help
553546
| Forward a port | `nemoclaw sandbox forward start <port> <name> -d` |
554547
| Create provider | `nemoclaw provider create --name N --type T --from-existing` |
555548
| List providers | `nemoclaw provider list` |
556-
| Create inference route | `nemoclaw inference create --routing-hint H --base-url U --model-id M` |
549+
| Configure cluster inference | `nemoclaw cluster inference set --provider P --model M` |
550+
| View cluster inference | `nemoclaw cluster inference get` |
557551
| Delete sandbox | `nemoclaw sandbox delete <name>` |
558552
| Destroy cluster | `nemoclaw cluster admin destroy` |
559553
| Self-teach any command | `nemoclaw <group> <cmd> --help` |

‎.agents/skills/nemoclaw-cli/cli-reference.md‎

Lines changed: 17 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,10 @@ nemoclaw
2828
│ ├── status
2929
│ ├── use <name>
3030
│ ├── list
31+
│ ├── inference
32+
│ │ ├── set --provider --model
33+
│ │ ├── update [--provider] [--model]
34+
│ │ └── get
3135
│ └── admin
3236
│ ├── deploy [opts]
3337
│ ├── stop [opts]
@@ -59,11 +63,6 @@ nemoclaw
5963
│ ├── list [opts]
6064
│ ├── update <name> --type [opts]
6165
│ └── delete <name>...
62-
├── inference
63-
│ ├── create [opts]
64-
│ ├── update <name> [opts]
65-
│ ├── delete <name>...
66-
│ └── list [opts]
6766
├── term
6867
├── completions <shell>
6968
└── ssh-proxy [opts]
@@ -302,38 +301,29 @@ Delete one or more providers by name.
302301

303302
---
304303

305-
## Inference Commands
304+
## Cluster Inference Commands
306305

307-
### `nemoclaw inference create`
306+
### `nemoclaw cluster inference set`
308307

309-
Create an inference route. Auto-detects supported protocols if `--protocol` is omitted.
308+
Configure the managed cluster inference route used by `inference.local`. Both flags are required.
310309

311310
| Flag | Default | Description |
312311
|------|---------|-------------|
313-
| `--name <NAME>` | auto-generated | Route name |
314-
| `--routing-hint <HINT>` | -- | Routing hint (required) |
315-
| `--base-url <URL>` | -- | Inference endpoint base URL (required) |
316-
| `--protocol <PROTO>` | auto-detected | Protocol(s): `openai_chat_completions`, `openai_completions`, `anthropic_messages`. Repeatable. |
317-
| `--api-key <KEY>` | `""` | API key for the endpoint |
318-
| `--model-id <ID>` | -- | Model identifier (required) |
319-
| `--disabled` | false | Create in disabled state |
320-
321-
### `nemoclaw inference update <name>`
322-
323-
Update an existing inference route. Same flags as `create`.
324-
325-
### `nemoclaw inference delete <NAME>...`
312+
| `--provider <NAME>` | -- | Provider record name (required) |
313+
| `--model <ID>` | -- | Model identifier to use for generation requests (required) |
326314

327-
Delete inference routes by name.
315+
### `nemoclaw cluster inference update`
328316

329-
### `nemoclaw inference list`
330-
331-
List inference routes.
317+
Partially update the cluster inference configuration. Fetches the current config and applies only the provided overrides. At least one flag is required.
332318

333319
| Flag | Default | Description |
334320
|------|---------|-------------|
335-
| `--limit <N>` | 100 | Max routes |
336-
| `--offset <N>` | 0 | Pagination offset |
321+
| `--provider <NAME>` | unchanged | Provider record name |
322+
| `--model <ID>` | unchanged | Model identifier |
323+
324+
### `nemoclaw cluster inference get`
325+
326+
Show the current cluster inference configuration.
337327

338328
---
339329

‎.claude/agent-memory/arch-doc-writer/MEMORY.md‎

Lines changed: 36 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,21 @@
11
# Arch Doc Writer Memory
22

33
## Project Structure
4-
- Crates: `navigator-cli`, `navigator-server`, `navigator-sandbox`, `navigator-bootstrap`, `navigator-core`, `navigator-providers`, `navigator-router`
4+
- Crates: `navigator-cli`, `navigator-server`, `navigator-sandbox`, `navigator-bootstrap`, `navigator-core`, `navigator-providers`, `navigator-router`, `navigator-policy`
55
- CLI entry: `crates/navigator-cli/src/main.rs` (clap parser + dispatch)
66
- CLI logic: `crates/navigator-cli/src/run.rs` (all command implementations)
77
- Sandbox entry: `crates/navigator-sandbox/src/lib.rs` (`run_sandbox()`)
88
- OPA engine: `crates/navigator-sandbox/src/opa.rs` (single file, not a directory)
99
- Identity cache: `crates/navigator-sandbox/src/identity.rs` (SHA256 TOFU, uses Mutex<HashMap> NOT DashMap)
1010
- L7 inspection: `crates/navigator-sandbox/src/l7/` (mod.rs, tls.rs, relay.rs, rest.rs, provider.rs, inference.rs)
1111
- Proxy: `crates/navigator-sandbox/src/proxy.rs`
12+
- Policy crate: `crates/navigator-policy/src/lib.rs` (YAML<->proto conversion, validation, restrictive default)
1213
- Server multiplex: `crates/navigator-server/src/multiplex.rs`
1314
- SSH tunnel: `crates/navigator-server/src/ssh_tunnel.rs`
1415
- Sandbox SSH server: `crates/navigator-sandbox/src/ssh.rs`
1516
- Providers: `crates/navigator-providers/src/providers/` (per-provider modules)
1617
- Bootstrap: `crates/navigator-bootstrap/src/lib.rs` (cluster lifecycle)
17-
- Proto files: `proto/` directory (navigator.proto, sandbox.proto, datamodel.proto)
18+
- Proto files: `proto/` directory (navigator.proto, sandbox.proto, datamodel.proto, inference.proto)
1819

1920
## Architecture Docs
2021
- Files renamed from numbered prefix format to descriptive names (e.g., `2 - server-architecture.md` -> `gateway-architecture.md`)
@@ -24,7 +25,9 @@
2425

2526
## Key Patterns
2627
- OPA baked-in rules: `include_str!("../data/sandbox-policy.rego")` in opa.rs
27-
- Policy loading: gRPC mode (NAVIGATOR_SANDBOX_ID + NAVIGATOR_ENDPOINT) or file mode (--policy-rules + --policy-data)
28+
- Policy loading: gRPC mode (NEMOCLAW_SANDBOX_ID + NEMOCLAW_ENDPOINT) or file mode (--policy-rules + --policy-data)
29+
- Env vars: sandbox uses NEMOCLAW_* prefix (e.g., NEMOCLAW_SANDBOX_ID, NEMOCLAW_ENDPOINT, NEMOCLAW_POLICY_RULES)
30+
- CLI flag: `--navigator-endpoint` (NOT `--nemoclaw-endpoint`)
2831
- Provider env injection: both entrypoint process (tokio Command) and SSH shell (std Command)
2932
- Cluster bootstrap: `sandbox_create_with_bootstrap()` auto-deploys when no cluster exists (main.rs ~line 632)
3033
- CLI cluster resolution: --cluster flag > NAVIGATOR_CLUSTER env > active cluster file
@@ -41,8 +44,9 @@
4144

4245
## Server Crate Details
4346
- Two gRPC services: Navigator (grpc.rs) and Inference (inference.rs), multiplexed via GrpcRouter by URI path
44-
- Gateway is control-plane only for inference: route CRUD + GetSandboxInferenceBundle
45-
- GetSandboxInferenceBundle: returns SandboxResolvedRoute list + revision hash + generated_at_ms for a sandbox_id
47+
- Gateway is control-plane only for inference: SetClusterInference + GetClusterInference + GetInferenceBundle
48+
- GetInferenceBundle: resolves managed route from provider record at request time, returns ResolvedRoute list + revision hash + generated_at_ms
49+
- SetClusterInference: takes provider_name + model_id, stores only references (endpoint/key/protocols resolved at bundle time)
4650
- Persistence: single `objects` table, protobuf payloads, Store enum dispatches SQLite vs Postgres by URL prefix
4751
- Persistence CRUD: upsert ON CONFLICT (id) not (object_type, id); list ORDER BY created_at_ms ASC, name ASC (not id!)
4852
- --db-url has no code default; Helm values.yaml sets `sqlite:/var/navigator/navigator.db`
@@ -83,10 +87,10 @@
8387
- Poll loop: `run_policy_poll_loop()` in lib.rs, spawned after child process, gRPC mode only
8488
- `OpaEngine::reload_from_proto()`: reuses `from_proto()` pipeline, atomically swaps inner engine, LKG on failure
8589
- `CachedNavigatorClient` in grpc_client.rs: persistent mTLS channel for poll + status report (mirrors CachedInferenceClient)
86-
- Dynamic domains: network_policies, inference (OPA engine swap). Static domains: filesystem, landlock, process (pre_exec, immutable)
90+
- Dynamic domains: network_policies only (inference removed from policy). Static domains: filesystem, landlock, process (pre_exec, immutable)
8791
- Server-side: `UpdateSandboxPolicy` RPC rejects changes to static fields or network mode changes
8892
- Server-side validation: `validate_static_fields_unchanged()` + `validate_network_mode_unchanged()` in grpc.rs
89-
- Poll interval: `NAVIGATOR_POLICY_POLL_INTERVAL_SECS` env var (default 30), no CLI flag
93+
- Poll interval: `NEMOCLAW_POLICY_POLL_INTERVAL_SECS` env var (default 30), no CLI flag
9094
- Version tracking: monotonic i64 per sandbox, `GetSandboxPolicyResponse` has version + policy_hash
9195
- Version 1 backfill: lazy on first `GetSandboxPolicy` from spec.policy if no policy_revisions row exists
9296
- `supersede_pending_policies()`: marks older pending revisions as superseded when new version persisted
@@ -99,22 +103,34 @@
99103
- CLI: `sandbox_policy_set()` in run.rs (~line 2901): loads YAML, calls UpdateSandboxPolicy, optionally polls for status
100104
- CLI: `sandbox_policy_get()` in run.rs (~line 3015): supports --rev N (version=0 means latest) and --full (YAML output via policy_to_yaml)
101105
- CLI: `sandbox_logs()` in run.rs (~line 3124): --source (all/gateway/sandbox) and --level (error/warn/info/debug/trace) filters
102-
- Deterministic hashing: `deterministic_policy_hash()` in grpc.rs (~line 1133): sorts network_policies by key, hashes fields individually
106+
- Deterministic hashing: `deterministic_policy_hash()` in grpc.rs (~line 1222): sorts network_policies by key, hashes fields individually, NO inference field
103107
- Idempotent UpdateSandboxPolicy: compares hash of new policy to latest stored hash, returns existing version if match
104-
- `policy_to_yaml()` in run.rs (~line 1623): converts proto to YAML via `PolicyYaml` struct (uses BTreeMap for ordered keys)
105-
- `policy_record_to_revision()` in grpc.rs (~line 1234): `include_policy` param controls whether full proto is included
108+
- `policy_to_yaml()` in run.rs: converts proto to YAML via navigator_policy::serialize_sandbox_policy (moved to navigator-policy crate)
109+
- `policy_record_to_revision()` in grpc.rs (~line 1334): `include_policy` param controls whether full proto is included
106110
- Server-side log filtering: `source_matches()` + `level_matches()` in grpc.rs, applied in both get_sandbox_logs and watch_sandbox
107-
- Standalone `proxy_inference()` was removed; proxy uses `CachedInferenceClient` via `InferenceContext.grpc_client` (OnceCell)
111+
- Standalone `proxy_inference()` was removed; inference handled in-sandbox by navigator-router
112+
- Provider types: claude, codex, opencode, generic, openai, anthropic, nvidia, gitlab, github, outlook
108113

109114
## Policy System Details
110-
- YAML data file top-level keys: filesystem_policy, landlock, process, network_policies, inference
115+
- YAML data file top-level keys: filesystem_policy, landlock, process, network_policies (NO inference key -- removed)
116+
- Proto SandboxPolicy fields: version, filesystem, landlock, process, network_policies (NO inference field)
111117
- Proto message field `filesystem` maps to YAML key `filesystem_policy` (different names!)
112-
- Behavioral trigger: network_policies non-empty -> proxy mode, empty -> block mode (seccomp blocks AF_INET/AF_INET6)
118+
- IMPORTANT: Sandbox always runs in Proxy mode. NetworkMode::Block exists as enum variant but is NEVER set.
119+
- Both file mode and gRPC mode set NetworkMode::Proxy unconditionally (see load_policy() in lib.rs and TryFrom in policy.rs)
120+
- Reason: proxy always needed so inference.local is addressable + all egress evaluated by OPA
121+
- OPA two-action model: Allow, Deny (NetworkAction in opa.rs). InspectForInference was REMOVED.
122+
- Rego network_action rule: "allow" or "deny" only (no "inspect_for_inference")
113123
- Behavioral trigger: endpoint `protocol` field -> L7 inspection; absent -> L4 raw copy_bidirectional
114124
- Behavioral trigger: `tls: terminate` -> MITM TLS with ephemeral CA; requires `protocol` to also be set
115125
- Behavioral trigger: `enforcement: enforce` -> deny at proxy; `audit` (default) -> log + forward
116126
- Access presets: read-only (GET/HEAD/OPTIONS), read-write (+POST/PUT/PATCH), full (*/*)
117127
- Validation: rules+access mutual exclusion, protocol requires rules/access, sql+enforce blocked, empty rules rejected
128+
- YAML policy parsing moved to navigator-policy crate (parse_sandbox_policy, serialize_sandbox_policy)
129+
- PolicyFile uses deny_unknown_fields for strict YAML parsing
130+
- restrictive_default_policy() in navigator-policy: no network policies, sandbox user, best_effort landlock
131+
- CONTAINER_POLICY_PATH: /etc/navigator/policy.yaml (well-known path for container-shipped policy)
132+
- clear_process_identity(): clears run_as_user/run_as_group for custom images
133+
- Policy safety validation: validate_sandbox_policy() checks root identity, path traversal, relative paths, overly broad paths, max 256 paths, max 4096 chars
118134
- Identity binding: /proc/net/tcp -> inode -> PID -> /proc/PID/exe + ancestors + cmdline, SHA256 TOFU cache
119135
- Network namespace: 10.200.0.1 (host/proxy) <-> 10.200.0.2 (sandbox), port 3128 default
120136
- Enforcement order in pre_exec: setns -> drop_privileges -> landlock -> seccomp
@@ -132,11 +148,14 @@
132148

133149
## Inference Routing Details
134150
- Sandbox-local execution via navigator-router crate
135-
- OPA three-action model: Allow, InspectForInference, Deny (`NetworkAction` in opa.rs)
136-
- InferenceContext: Router + patterns + `Arc<RwLock<Vec<ResolvedRoute>>>` route cache
151+
- InferenceContext in proxy.rs: Router + patterns + `Arc<RwLock<Vec<ResolvedRoute>>>` route cache
137152
- Route sources: `--inference-routes` YAML file (standalone) > cluster bundle via gRPC; empty routes gracefully disable
138153
- Cluster bundle refreshed every ROUTE_REFRESH_INTERVAL_SECS (30s)
139-
- Patterns: POST /v1/chat/completions, /v1/completions, /v1/responses, /v1/messages
154+
- Patterns: POST /v1/chat/completions, /v1/completions, /v1/responses, /v1/messages; GET /v1/models, /v1/models/*
155+
- inference.local CONNECT intercepted BEFORE OPA evaluation in proxy
156+
- InferenceProviderProfile in navigator-core/src/inference.rs: centralized provider metadata
157+
- proxy.rs: ONLY CONNECT to inference.local is handled; non-CONNECT requests get 403 for ALL hosts
158+
- Buffer: INITIAL_INFERENCE_BUF=64KiB, MAX_INFERENCE_BUF=10MiB; grows by doubling
140159
- Dev sandbox: `mise run sandbox -e VAR_NAME` forwards host env vars; NVIDIA_API_KEY always passed
141160

142161
## Log Streaming Details
@@ -162,4 +181,4 @@
162181
## Naming Conventions
163182
- The project name "Navigator" appears in code but docs should use generic terms per user preference
164183
- CLI binary: `navigator` (aliased as `nav` in dev via mise)
165-
- Provider types: claude, codex, opencode, openclaw, generic, nvidia, gitlab, github, outlook
184+
- Provider types: claude, codex, opencode, generic, openai, anthropic, nvidia, gitlab, github, outlook (see ProviderRegistry::new())

‎Cargo.lock‎

Lines changed: 10 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)