refactor(pipelines): move the pipeline engine to @deepnote/pipelines

The engine imports no `node:*` by design, so a pipeline can run in a browser tab. That made `@deepnote/local-runner` — a local, Python-backed runner — the wrong home for it. Move it to its own package before the API is published, and name it after what it does. - `orchestrate` -> `runPipeline`, `runOrchestration` -> `runPipelineWithExecutor`, `Orchestration*` -> `Pipeline*`, `orchestrationOutputs` -> `pipelineOutputs`. - Snapshot reading moves too: `snapshot-view.ts` and `input-info.ts` are the browser-safe half of local-runner and the pipeline needs them, so keeping them behind would have made the two packages depend on each other. `local-runner` re-exports them, and `local-runner/snapshot-reader` still bundles them, so nothing already published changes shape. - `RunBlockOutput` moves alongside, with `ExecutionSummary` and `AgentStreamEvent` restated rather than imported: depending on `@deepnote/runtime-core` for two type aliases would pull a Python execution engine into a browser bundle. `pipeline-types.test.ts` fails to compile if the shapes drift. - The example moves to `examples/pipelines/script`, run by `pnpm example:pipeline`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jamesbhobbs · Aug 27, 2026 · Aug 27, 2026 · Aug 31, 2026 · Sep 2, 2026 · Sep 2, 2026
commit b300446fb5964ced70aec98e792a8d8595ef491a
@@ -13,6 +13,7 @@ This is a TypeScript monorepo for Deepnote's open-source packages, managed with
 - **packages/database-integrations** - Database integration definitions, schemas, and authentication methods
 - **packages/local-runner** - Local Python-backed runner and static UI for Deepnote notebooks
 - **packages/mcp** - MCP server for AI-assisted Deepnote notebook creation and manipulation
+- **packages/pipelines** - Compose Deepnote notebook runs into pipelines, with no server and no orchestration engine
 - **packages/reactivity** - Reactivity and dependency graph for Deepnote notebooks
 - **packages/runtime-core** - Core runtime for executing Deepnote projects
 
@@ -28,6 +29,7 @@ Start with the owning package and its README before searching broadly. Avoid tra
 | MCP tools and resources                                        | `packages/mcp/`                                       |
 | Deepnote Cloud runs and schedules API clients                  | `packages/cloud/`                                     |
 | Local notebook execution and serving                           | `packages/local-runner/` and `packages/runtime-core/` |
+| Composing several notebook runs into one pipeline              | `packages/pipelines/`                                 |
 | Dependency and reactivity analysis                             | `packages/reactivity/`                                |
 | Database integration definitions                               | `packages/database-integrations/`                     |
 | Shared test data                                               | `test-fixtures/`                                      |

@@ -1,15 +1,15 @@
-# orchestration
+# A pipeline as a script
 
 A pipeline as a script: fan out across regional notebooks, gate on their structured results, and
 report. About 40 lines.
 
 ```bash
-DEEPNOTE_TOKEN=… NA_NOTEBOOK_ID=… EU_NOTEBOOK_ID=… APAC_NOTEBOOK_ID=… pnpm example:orchestration
+DEEPNOTE_TOKEN=… NA_NOTEBOOK_ID=… EU_NOTEBOOK_ID=… APAC_NOTEBOOK_ID=… pnpm example:pipeline
 ```
 
-The point of the example is that the _pipeline_ is not Node-specific. `orchestrate` runs every step
+The point of the example is that the _pipeline_ is not Node-specific. `runPipeline` runs every step
 as an HTTP call to Deepnote, so the callback below runs unchanged in a browser page with no server
-behind it — see [`run-app`](../run-app), which uses the same API.
+behind it — see [`run-app`](../../local-runner/run-app), which uses the same API.
 
 The bootstrap is Node and would be replaced in a browser: this script reads `DEEPNOTE_TOKEN` and the
 notebook ids from the environment via `process`, whereas a page gets its token from the Deepnote

@@ -1,12 +1,12 @@
 // A pipeline in ~40 lines: fan out, gate on the results, then decide.
 //
 // The bootstrap below is Node — `process.loadEnvFile`, `process.env`, `process.exit`. The pipeline
-// itself is not: the callback passed to `orchestrate` runs unchanged in a browser, because every
+// itself is not: the callback passed to `runPipeline` runs unchanged in a browser, because every
 // step is an HTTP call to Deepnote. A page supplies its own configuration and token (see
 // `examples/local-runner/run-app`); what it does not need is a server, a kernel, or a daemon.
 //
-// In a real project, import from '@deepnote/local-runner' after installing it.
-import { orchestrate } from '../../../packages/local-runner/dist/index.js'
+// In a real project, import from '@deepnote/pipelines' after installing it.
+import { runPipeline } from '../../../packages/pipelines/dist/index.js'
 
 try {
   process.loadEnvFile()
@@ -32,7 +32,7 @@ if (REGIONS.length === 0) {
 
 const QUALITY_THRESHOLD = 0.95
 
-const { value, graph, durationMs } = await orchestrate(
+const { value, graph, durationMs } = await runPipeline(
   async ({ run, control, outputs }) => {
     // Ordinary control flow is the pipeline: these fan out concurrently because Promise.all does.
     const analyses = await Promise.all(

@@ -17,7 +17,7 @@
     "build": "pnpm -r run build",
     "example:gallery": "pnpm --filter @deepnote/local-runner... build && node examples/local-runner/gallery/serve.mjs",
     "example:local-runner": "pnpm --filter @deepnote/local-runner... build && node examples/local-runner/run-app/serve.mjs",
-    "example:orchestration": "pnpm --filter @deepnote/local-runner... build && node examples/local-runner/orchestration/run.mjs",
+    "example:pipeline": "pnpm --filter @deepnote/pipelines... build && node examples/pipelines/script/run.mjs",
     "example:schedule-cloud": "pnpm --filter @deepnote/local-runner... build && node examples/local-runner/schedule-cloud.mjs",
     "example:snapshot-viewer": "pnpm --filter @deepnote/local-runner... build && node examples/local-runner/snapshot-viewer/serve.mjs",
     "license-check": "license-checker-rseidelsohn --json --onlyAllow \"MIT;Apache-2.0;BSD-2-Clause;BSD-3-Clause;ISC\" --excludePackages \"deepnote\"",

@@ -183,64 +183,9 @@ The server binds to `127.0.0.1` and provides no WebSocket, watch, or rendering.
 — or, to _view_ an existing snapshot rather than run one, read it directly (below); that needs no
 server at all.
 
-### Orchestrate notebook pipelines
-
-Run several notebooks as one pipeline — fan out, gate on the results, decide:
-
-```ts
-import { orchestrate } from "@deepnote/local-runner";
-
-const { value, graph } = await orchestrate(
-  async ({ run, control, outputs }) => {
-    const analyses = await Promise.all(
-      REGIONS.map((region) =>
-        run({
-          id: region.name,
-          notebookId: region.notebookId,
-          inputs: { region: region.name },
-        }),
-      ),
-    );
-    const readings = analyses.map((step) => outputs.lastJson(step));
-
-    const failing = await control(
-      {
-        id: "quality-gate",
-        kind: "gate",
-        dependsOn: analyses.map((s) => s.id),
-      },
-      () => readings.filter((r) => r.qualityScore < 0.95).map((r) => r.region),
-    );
-
-    return { checked: readings.length, failing };
-  },
-  { token, onEvent: (event) => render(event) },
-);
-```
-
-This is deliberately an imperative API, not a workflow language: `await`, `Promise.all`, loops and
-branches in the callback provide sequencing, concurrency, and conditionals. The library records what
-happened — the graph, the events, the normalized results.
-
-**It needs no server and no local kernel.** Every step is an HTTP call to Deepnote, so the same
-pipeline runs in a script, in CI, and in a browser page. Nothing reachable from `orchestrate`
-imports `node:*`.
-
-Notebooks are addressed by id and must already exist. Running a pipeline needs permission to run a
-notebook, not to create one — which is what lets a page do it with a viewer's short-lived token.
-
-`control` records a local decision as a node, so a gate or an aggregation shows up in the graph
-instead of happening invisibly between steps. `outputs.lastJson(step)` and
-`outputs.lastAgentText(step)` read a step's results without depending on block ids, which Deepnote
-reassigns when it creates a notebook.
-
-A failed notebook throws `OrchestrationStepError` carrying the result, so a caller can still show
-how far the run got; `allowFailure: true` returns it instead.
-
-`runOrchestration(workflow, options, executor)` is the same engine with the runner left open, for
-callers that want to run steps somewhere else.
-
-See [`examples/local-runner/orchestration`](../../examples/local-runner/orchestration).
+To run several notebooks as one pipeline — fanning out, gating on results, drawing the graph — see
+[`@deepnote/pipelines`](../pipelines). That package needs neither this one nor a kernel: every step
+is an HTTP call, so a pipeline also runs in a browser page.
 
 ### Read a snapshot — no Python, no kernel
 

@@ -41,6 +41,7 @@
     "@deepnote/blocks": "workspace:*",
     "@deepnote/cloud": "workspace:*",
     "@deepnote/convert": "workspace:*",
+    "@deepnote/pipelines": "workspace:*",
     "@deepnote/runtime-core": "workspace:*",
     "@jupyterlab/nbformat": "^4.6.3"
   },

@@ -1,10 +1,10 @@
 import type { DeepnoteFile, InputBlock, InputBlockValueOverride, InputBlockValueOverrides } from '@deepnote/blocks'
+import type { InputBlockInfo } from '@deepnote/pipelines'
+import { inputInfoFor } from '@deepnote/pipelines'
 import type { InputScope } from './coerce-input-value'
 import { coerceInputValueForBlocks, inputBlocksByName, notebooksInScope } from './coerce-input-value'
-import type { InputBlockInfo } from './input-info'
-import { inputInfoFor } from './input-info'
 
-export type { InputBlockInfo } from './input-info'
+export type { InputBlockInfo } from '@deepnote/pipelines'
 
 /**
  * Apply input overrides to a file's input blocks, in place.

@@ -9,6 +9,11 @@
  * produces (how a table looks, whether HTML output is sandboxed) belong to the page, not the
  * library. See `examples/local-runner/snapshot-viewer` for a complete one.
  */
-export type { InputBlockInfo } from './input-info'
-export type { SnapshotBlock, SnapshotInput, SnapshotNotebook, SnapshotView } from './snapshot-view'
-export { parseSnapshot, toSnapshotView } from './snapshot-view'
+export type {
+  InputBlockInfo,
+  SnapshotBlock,
+  SnapshotInput,
+  SnapshotNotebook,
+  SnapshotView,
+} from '@deepnote/pipelines'
+export { parseSnapshot, toSnapshotView } from '@deepnote/pipelines'
@@ -1,9 +1,9 @@
 import type { DeepnoteFile } from '@deepnote/blocks'
 import { deepnoteFileSchema, deepnoteSnapshotSchema, parseYaml } from '@deepnote/blocks'
 import { describeRunError, findNotebook, getWorkspace, type NormalizedRun, notebookUrl } from '@deepnote/cloud'
-import { parseSnapshot } from './snapshot-view'
+import { parseSnapshot } from '@deepnote/pipelines'
 
-export { extractOutputs } from './extract-outputs'
+export { extractOutputs } from '@deepnote/pipelines'
 
 /**
  * The plumbing every cloud entry point needs — `run-in-cloud.ts` and `cloud-runs.ts` both reach for

@@ -1,9 +1,9 @@
+export type { SnapshotBlock, SnapshotInput, SnapshotNotebook, SnapshotView } from '@deepnote/pipelines'
+export { parseSnapshot, toSnapshotView } from '@deepnote/pipelines'
 export type { AgentStreamEvent } from '@deepnote/runtime-core'
 export type { InputBlockInfo } from './apply-input-overrides'
 export { applyInputOverrides, listInputBlocks } from './apply-input-overrides'
 export { mapBlockIds, toBlockSpec } from './block-spec'
-export type { CloudExecutorOptions } from './cloud-executor'
-export { createCloudStepExecutor, DEFAULT_CLOUD_API_URL, toRunInputs } from './cloud-executor'
 export type {
   CloudRun,
   GetCloudRunOptions,
@@ -15,38 +15,6 @@ export type { DeepnoteInput, LoadedDeepnoteFile } from './load-file'
 export { loadDeepnoteFile } from './load-file'
 export type { OpenInCloudOptions } from './open-in-cloud'
 export { openInCloud } from './open-in-cloud'
-export type {
-  OrchestrateOptions,
-  OrchestrationContext,
-  OrchestrationControlKind,
-  OrchestrationControlNode,
-  OrchestrationDependency,
-  OrchestrationDependencyInput,
-  OrchestrationEvent,
-  OrchestrationGraph,
-  OrchestrationGraphEdge,
-  OrchestrationGraphNode,
-  OrchestrationGraphNodeKind,
-  OrchestrationGraphNodeStatus,
-  OrchestrationOutputHelpers,
-  OrchestrationResult,
-  OrchestrationStep,
-  OrchestrationStepExecution,
-  OrchestrationStepExecutor,
-  OrchestrationStepResult,
-} from './orchestrate'
-export {
-  allOutputText,
-  finishResult,
-  lastAgentText,
-  lastOutputJson,
-  OrchestrationStepError,
-  orchestrate,
-  orchestrationOutputs,
-  outputJson,
-  outputText,
-  runOrchestration,
-} from './orchestrate'
 export { readSnapshot } from './read-snapshot'
 export type { RecurringSchedule, ResolvedRecurringSchedule } from './recurring-schedule'
 export { RecurringScheduleError, resolveRecurringSchedule } from './recurring-schedule'
@@ -66,8 +34,6 @@ export type {
   ServeStaticOptions,
 } from './serve-static'
 export { serveStatic } from './serve-static'
-export type { SnapshotBlock, SnapshotInput, SnapshotNotebook, SnapshotView } from './snapshot-view'
-export { parseSnapshot, toSnapshotView } from './snapshot-view'
 export type {
   BlockMove,
   DetailedSyncPlan,

diff --git a/packages/local-runner/src/pipeline-types.test.ts b/packages/local-runner/src/pipeline-types.test.ts
@@ -0,0 +1,32 @@
+import type {
+  AgentStreamEvent as PipelineAgentStreamEvent,
+  ExecutionSummary as PipelineExecutionSummary,
+} from '@deepnote/pipelines'
+import type { AgentStreamEvent, ExecutionSummary } from '@deepnote/runtime-core'
+import { describe, expect, it } from 'vitest'
+
+/**
+ * `@deepnote/pipelines` restates these two types rather than depending on `@deepnote/runtime-core`,
+ * so that composing cloud runs in a browser does not pull in a Python execution engine. That is
+ * only safe while the shapes stay assignable in both directions, which is what this asserts: a
+ * local run's summary can be reported as a pipeline step's, and vice versa.
+ *
+ * If a field is added to one side, this stops compiling — which is the point.
+ */
+describe('pipeline types mirror the runtime-core types', () => {
+  it('keeps ExecutionSummary assignable in both directions', () => {
+    const fromRuntime: ExecutionSummary = { totalBlocks: 3, executedBlocks: 3, failedBlocks: 0, totalDurationMs: 12 }
+    const asPipeline: PipelineExecutionSummary = fromRuntime
+    const backAgain: ExecutionSummary = asPipeline
+
+    expect(backAgain).toEqual(fromRuntime)
+  })
+
+  it('keeps AgentStreamEvent assignable in both directions', () => {
+    const fromRuntime: AgentStreamEvent = { type: 'text_delta', text: 'hello' }
+    const asPipeline: PipelineAgentStreamEvent = fromRuntime
+    const backAgain: AgentStreamEvent = asPipeline
+
+    expect(backAgain).toEqual(fromRuntime)
+  })
+})
@@ -0,0 +1,27 @@
+import { mkdtempSync, readFileSync, writeFileSync } from 'node:fs'
+import { tmpdir } from 'node:os'
+import path from 'node:path'
+import { deserializeDeepnoteFile } from '@deepnote/blocks'
+import { describe, expect, it } from 'vitest'
+import { readSnapshot } from './read-snapshot'
+
+const FIXTURES = path.join(__dirname, '../../../test-fixtures')
+const SNAPSHOT = readFileSync(path.join(FIXTURES, 'snapshot-view.snapshot.deepnote'), 'utf8')
+
+/**
+ * The parsing itself is tested in `@deepnote/pipelines`, which owns it. What is left here is the
+ * part that needs Node: deciding whether a string is a path or the snapshot itself.
+ */
+describe('readSnapshot', () => {
+  it('reads from a path, raw YAML, or a parsed object', () => {
+    const dir = mkdtempSync(path.join(tmpdir(), 'snap-'))
+    const snapshotPath = path.join(dir, 'run.snapshot.deepnote')
+    writeFileSync(snapshotPath, SNAPSHOT)
+
+    expect(readSnapshot(snapshotPath).projectName).toBe('Sales')
+    expect(readSnapshot(SNAPSHOT).projectName).toBe('Sales')
+
+    const file = deserializeDeepnoteFile(SNAPSHOT)
+    expect(readSnapshot(file).notebooks[0].blocks.map(b => b.id)).toEqual(['b-input', 'b-md', 'b-code', 'b-sql'])
+  })
+})
@@ -1,7 +1,7 @@
 import { existsSync, readFileSync } from 'node:fs'
 import type { DeepnoteFile } from '@deepnote/blocks'
-import type { SnapshotView } from './snapshot-view'
-import { parseSnapshot, toSnapshotView } from './snapshot-view'
+import type { SnapshotView } from '@deepnote/pipelines'
+import { parseSnapshot, toSnapshotView } from '@deepnote/pipelines'
 
 /**
  * Read a `.deepnote` snapshot from a path, raw YAML, or an already-parsed file.

@@ -3,6 +3,7 @@ import type { DeepnoteFile, DeepnoteSnapshot } from '@deepnote/blocks'
 import { serializeDeepnoteSnapshot } from '@deepnote/blocks'
 import type { BlockExecutionOutput } from '@deepnote/convert'
 import { mergeOutputsIntoFile, saveExecutionSnapshot, splitDeepnoteFile } from '@deepnote/convert'
+import type { RunBlockOutput } from '@deepnote/pipelines'
 import type { AgentStreamEvent, BlockExecutionResult, ExecutionSummary, IOutput } from '@deepnote/runtime-core'
 import { detectDefaultPython, ExecutionEngine } from '@deepnote/runtime-core'
 import { applyInputOverrides } from './apply-input-overrides'
@@ -36,11 +37,11 @@ export interface RunWithInputsOptions {
   onAgentEvent?: (event: AgentStreamEvent) => void | Promise<void>
 }
 
-export interface RunBlockOutput {
-  blockId: string
-  outputs: IOutput[]
-  executionCount: number | null
-}
+/**
+ * Re-exported so a caller does not have to know whether a run happened here or in the cloud: a
+ * local run and a pipeline step describe their block outputs with the same type.
+ */
+export type { RunBlockOutput } from '@deepnote/pipelines'
 
 export interface RunWithInputsResult {
   /** Per-block outputs, in execution order. */

@@ -6,7 +6,13 @@ export default defineConfig([
     format: ['esm', 'cjs'],
     fixedExtension: false,
     dts: true,
-    external: ['@deepnote/blocks', '@deepnote/cloud', '@deepnote/convert', '@deepnote/runtime-core'],
+    external: [
+      '@deepnote/blocks',
+      '@deepnote/cloud',
+      '@deepnote/convert',
+      '@deepnote/pipelines',
+      '@deepnote/runtime-core',
+    ],
   },
   {
     // The snapshot reader ships as one self-contained file that a static page can <script> in, so