Add OpenRouter samples: prompt batch and budget gate by DABH · Pull Request #366 · temporalio/samples-python

DABH · 2026-09-11T06:16:32Z

Adds an openrouter sample with two scenarios, plus OpenRouter as a model provider for the OpenAI Agents SDK sample.

prompt_batch fans one Activity out per prompt with OpenRouter's Auto Router and returns answer, model, and reported cost per prompt. The Activity uses the openai client pointed at OpenRouter with max_retries=0, classifies 4xx as non-retryable and 408/429/5xx as retryable, passes Retry-After through as next_retry_delay, heartbeats, and sends X-OpenRouter-Cache: true so a retried identical request is served from OpenRouter's cache at $0. A --fail-once flag demonstrates that.

budget_gate is the same batch, but it pauses on a soft budget or on OpenRouter's 402 and resumes on a raise_budget Update, with a spend_report Query.

Tests use httpx.MockTransport and a fake Activity; no API key needed. Verified live against OpenRouter, including a cache hit at $0 on retry.

Docs page: temporalio/documentation#5307. TypeScript port: temporalio/samples-typescript#520.

Copilot

🟡 Changes recommended

Input deadlocks, serialized approval timeouts, duplicate-prompt state loss, and incomplete cost accounting need resolution.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds OpenRouter integrations for durable prompt batching, budget-aware execution, and the OpenAI Agents SDK.

Changes:

Adds prompt-batch and budget-gate workflows with retries, caching, and cost reporting.
Adds mocked integration tests and dependency configuration.
Adds an OpenRouter-backed Agents SDK example and documentation.

File summaries

File	Description
`uv.lock`	Locks OpenRouter dependencies.
`pyproject.toml`	Adds the OpenRouter dependency group.
`README.md`	Registers the new sample.
`openrouter/__init__.py`	Initializes the package.
`openrouter/shared.py`	Defines shared inputs, results, and constants.
`openrouter/activities.py`	Implements OpenRouter requests and error handling.
`openrouter/README.md`	Documents the integration.
`openrouter/prompt_batch/__init__.py`	Initializes the prompt-batch package.
`openrouter/prompt_batch/workflow.py`	Implements concurrent prompt processing.
`openrouter/prompt_batch/run_worker.py`	Runs the prompt-batch worker.
`openrouter/prompt_batch/run_workflow.py`	Starts prompt-batch workflows.
`openrouter/prompt_batch/README.md`	Documents prompt batching.
`openrouter/budget_gate/__init__.py`	Initializes the budget-gate package.
`openrouter/budget_gate/workflow.py`	Implements budget reservations and pauses.
`openrouter/budget_gate/run_worker.py`	Runs the budget-gate worker.
`openrouter/budget_gate/run_workflow.py`	Starts budget-gate workflows.
`openrouter/budget_gate/raise_budget.py`	Sends budget updates.
`openrouter/budget_gate/README.md`	Documents budget controls.
`tests/openrouter/__init__.py`	Initializes the test package.
`tests/openrouter/activity_test.py`	Tests HTTP handling and caching.
`tests/openrouter/prompt_batch_test.py`	Tests batch result collection.
`tests/openrouter/budget_gate_test.py`	Tests pauses, updates, and timeouts.
`openai_agents/model_providers/workflows/openrouter_workflow.py`	Adds an OpenRouter agent workflow.
`openai_agents/model_providers/run_openrouter_workflow.py`	Starts the agent workflow.
`openai_agents/model_providers/run_openrouter_worker.py`	Configures the OpenRouter provider.
`openai_agents/model_providers/README.md`	Documents the provider example.

Review details

Suppressed comments (1)

openrouter/activities.py:192

This simulated successful first call is billed, but its result and cost are discarded when the Activity raises. The cached retry then reports $0, so BatchResult.total_cost_usd and the budget ledger say zero even though the account was charged; the same gap occurs during the real post-response Worker crash being demonstrated. Persist/reconcile the first generation's charge externally, or clearly expose these totals as lower bounds and document that limitation for the budget gate.

        if request.fail_once_after_call and activity.info().attempt == 1:
            # Demo hook: the Worker "crashes" after the response arrived. The
            # retry re-sends the identical request and gets a cache hit.
            raise ApplicationError(
                "Simulated failure after the response was received",
                type="SimulatedFailure",
            )

Files reviewed: 21/26 changed files
Comments generated: 5
Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Call OpenRouter from Activities with Temporal-owned retries, error classification, Retry-After handling, heartbeats, and OpenRouter response caching so a retried request is billed at zero. The budget_gate sample pauses the batch on a soft budget or on OpenRouter's 402 and resumes on a raise_budget Update. Also adds OpenRouter as a model provider for the OpenAI Agents SDK plugin sample.

Exceeding an OpenRouter API key's credit limit returns 403 "Key limit exceeded", not 402; 402 is the account running out of credits. The Activity now raises OpenRouterOutOfCredits for both so the budget gate pauses on either. Verified live with a key limit set below usage.

…uter for the agent - Reject max_concurrency < 1 and non-positive estimates before starting a batch instead of parking forever. - Parked prompts wait until one batch-wide deadline rather than each starting a fresh approval timeout. - The Agents sample defaults to openrouter/auto, and its in-Workflow tool is async: the Agents SDK runs sync tools in a thread, which the sandbox forbids.

Copilot

Copilot review overview

🟡 Changes recommended

Retry timing, duplicate-prompt tracking, non-finite budgets, and cost reporting have unresolved correctness issues.

Review effort: Balanced
Findings: 1 · 4 · 1

Open (6)

Treating an absent cost as 0.0 makes an unknown charge indistinguishable from a genuinely free…
Parse Retry-After HTTP-date values · New
Reject non-finite monetary estimates and budgets · New
Clarify total cost excludes discarded billed attempts · New
Using prompt text as the state key loses parked entries when the input contains duplicate prompts.…
Clarify spend bound excludes discarded billed attempts · New

Resolved since last review (3)

Semaphore(0) never admits any task, so a caller passing max_concurrency=0 (which the CLI…
The semaphore is held while _reserve waits for approval. Once the budget is exhausted, only…
The budget input is not validated before constructing the semaphore/reservation state.…

…wn cost, reported cost - Parse the HTTP-date form of Retry-After as well as delta-seconds. - Require finite budget and estimate values (argparse accepts inf and nan). - Rethrow Workflow cancellation instead of recording it as a skipped prompt. - A response without usage.cost yields cost_usd=None; the budget gate charges the estimate for it instead of treating the call as free. - The batch total is named reported_cost_usd and the READMEs say what it does and does not include.

brianstrauch

The concurrent budget-reservation issue below needs to be fixed before merging. All 17 existing tests and a mocked Agents weather-tool round trip passed; the concurrent-budget reproduction failed.

…, per-attempt wording - Parked prompts re-check the budget after waking. Raising the budget wakes every parked prompt before any runs, so without the re-check several could reserve the same headroom. Test added that reproduces the race. - BudgetGateInput now embeds BatchInput instead of redeclaring its fields. - READMEs and comments describe what Event History records (attempt count and last failure) versus what the Worker logs per attempt.

…ncellation class - Budget and deadline are set in @workflow.init so an update-with-start raise_budget is honored instead of being overwritten by run(). - A 402 whose error.metadata.limit_source is openrouter_in_flight_budget is transient per OpenRouter's docs; retry it after Retry-After. - A 200 with a partial answer and an error on the choice is an error, not an answer. - openai's APIError.body is already the inner error object, so messages were raw reprs; unwrap it and assert exact messages in tests. - Catch temporalio.exceptions.CancelledError, not asyncio's; the cancellation branch was dead code. Cancellation tests added for both workflows. - Shared retry policy moved to shared.py; paused_reason dropped; Agents sample imports temporalio.openai_agents like its siblings; README output regenerated to match the command shown; overshoot bound stated correctly.

…y responses - approval_timeout_seconds is validated in @workflow.init (0..30 days) so a bad value fails the Workflow instead of wedging its first task. - Retry-After is honored up to five minutes; next_retry_delay overrides the retry policy, so an unbounded value could park a prompt for days. - A 200 with no choices and no error is retried, not recorded as an answer. - Cancellation tests assert that no prompt was logged as skipped, which fails against the old asyncio.CancelledError check. - budget_gate README out-of-credits example matches the command shown; ledger numbers add up; comments on reported cost and paused keys fixed.

brianstrauch

Two reproducible issues in the budget gate. All 29 existing sample tests and a mocked Agents tool round trip with replay enabled passed; both additional reproductions failed.

brianstrauch · 2026-10-01T20:25:13Z

+    # @@@SNIPEND
+
+    async def _wait_for_more_credits(self, prompt: str) -> bool:
+        seen = self._budget_version


[P2] Preserve credit updates received while an Activity is in flight. seen is captured only after the credits failure reaches the Workflow. With two concurrent requests that both encountered insufficient credits, one can fail and park, the operator can top up and send raise_budget, and the second request's earlier failure can arrive afterward. That second prompt snapshots the already-incremented version and waits for another Update, eventually skipping if none arrives despite credits being available. I reproduced this: after one Update, a completed but b remained paused with insufficient_credits; a second identical Update released it. Capture the budget version before scheduling each Activity and compare against that version when handling its credits failure.

brianstrauch · 2026-10-01T20:25:13Z

+        """Reserve `estimate` against the budget, parking until it fits."""
+
+        def fits() -> bool:
+            return self._spent_usd + self._reserved_usd + estimate <= self._budget_usd


[P2] Avoid rejecting affordable calls because of floating-point accumulation. The strict comparison uses the unrounded sum of previous costs. I reproduced this with ten prompts, max_concurrency=1, budget_usd=0.01, and both the estimate and each actual cost equal to 0.001: only nine prompts ran. The tenth was rejected because the accumulated spend plus its estimate was 0.010000000000000002, greater than the 0.01 budget. With the normal approval timeout it unnecessarily parks; with a zero timeout it is skipped as soft_budget_exhausted. Use decimal/fixed-point accounting or a bounded comparison tolerance, and cover the exact-budget boundary in a regression test.

DABH requested review from a team as code owners September 11, 2026 06:16

This was referenced Sep 11, 2026

Add OpenRouter integration page (Python) temporalio/documentation#5307

Open

Add OpenRouter sample: prompt batch with cached retries temporalio/samples-typescript#520

Open

patbeqo reviewed Sep 11, 2026

View reviewed changes

Comment thread openai_agents/model_providers/workflows/openrouter_workflow.py Outdated

brianstrauch requested a balanced review from Copilot September 11, 2026 16:06

Copilot started reviewing on behalf of brianstrauch September 11, 2026 16:07 View session

Copilot AI reviewed Sep 11, 2026

View reviewed changes

Comment thread openrouter/activities.py Outdated

Comment thread openrouter/budget_gate/workflow.py Outdated

Comment thread openrouter/budget_gate/workflow.py

Comment thread openrouter/budget_gate/workflow.py Outdated

Comment thread openrouter/prompt_batch/workflow.py

DABH added 3 commits September 30, 2026 13:21

DABH force-pushed the dabh/openrouter branch from e7ce0b5 to 6595f89 Compare September 30, 2026 18:44

DABH requested a balanced review from Copilot October 1, 2026 06:09

Copilot started reviewing on behalf of DABH October 1, 2026 06:09 View session

Copilot AI reviewed Oct 1, 2026

View reviewed changes

Comment thread openrouter/activities.py

Comment thread openrouter/budget_gate/workflow.py Outdated

Comment thread openrouter/prompt_batch/run_workflow.py Outdated

Comment thread openrouter/budget_gate/README.md Outdated

brianstrauch requested changes Oct 1, 2026

View reviewed changes

Comment thread openrouter/budget_gate/workflow.py

JasonSteving99 reviewed Oct 1, 2026

View reviewed changes

Comment thread openrouter/shared.py

DABH requested a review from brianstrauch October 1, 2026 18:12

DABH added 2 commits October 1, 2026 13:27

brianstrauch reviewed Oct 1, 2026

View reviewed changes

Navigation Menu

Add OpenRouter samples: prompt batch and budget gate - #366

Add OpenRouter samples: prompt batch and budget gate#366
DABH wants to merge 7 commits into
mainfrom
dabh/openrouter

DABH commented Sep 11, 2026 •

edited

Loading

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

brianstrauch left a comment

Uh oh!

Uh oh!

Uh oh!

brianstrauch left a comment

Uh oh!

brianstrauch Oct 1, 2026

Uh oh!

brianstrauch Oct 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

DABH commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (1)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

brianstrauch left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

brianstrauch left a comment

Choose a reason for hiding this comment

Uh oh!

brianstrauch Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

brianstrauch Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

DABH commented Sep 11, 2026 •

edited

Loading