Repository navigation
feat(pipelines): compose Deepnote notebook runs into pipelines #496
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from 1 commit
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
b587099
feat(local-runner): orchestrate notebook pipelines
jamesbhobbs 95d6d5d
fix(local-runner): address review on the orchestration engine
jamesbhobbs b300446
refactor(pipelines): move the pipeline engine to @deepnote/pipelines
jamesbhobbs b0f5afa
refactor(pipelines): trim the public surface and drop dead result fields
jamesbhobbs 713dedd
feat(pipelines): cap concurrent notebook steps
jamesbhobbs 11a13d1
fix(cloud): wake the poll sleep when the run is aborted
jamesbhobbs fed8d12
feat(pipelines): send Date inputs as ISO 8601
jamesbhobbs 7988c4f
refactor(local-runner): take DEFAULT_CLOUD_API_URL from @deepnote/pip…
jamesbhobbs ec57a40
docs(pipelines): fix stale references in comments and the example
jamesbhobbs ca8ccc5
feat(pipelines): keep the partial run when a pipeline fails
jamesbhobbs 7b3dfd6
fix(pipelines): let allowFailure cover a throwing executor
jamesbhobbs File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| # orchestration | ||
|
|
||
| A pipeline as a script: fan out across regional notebooks, gate on their structured results, and | ||
| report. About 40 lines. | ||
|
|
||
| ```bash | ||
| DEEPNOTE_TOKEN=… NA_NOTEBOOK_ID=… EU_NOTEBOOK_ID=… APAC_NOTEBOOK_ID=… pnpm example:orchestration | ||
| ``` | ||
|
|
||
| The point of the example is that none of it is Node-specific. `orchestrate` runs every step as an | ||
| HTTP call to Deepnote, so the same pipeline runs unchanged in a browser page with no server behind | ||
| it — see [`run-app`](../run-app). What Node adds here is a shell, not a capability. | ||
|
|
||
| Notebooks are named by id and must already exist: running a pipeline needs permission to run a | ||
| notebook, not to create one. | ||
|
|
||
| Each regional notebook should end by emitting a JSON object the gate can read: | ||
|
|
||
| ```python | ||
| print(json.dumps({"region": region, "revenueK": 812.4, "qualityScore": 0.97})) | ||
| ``` | ||
|
|
||
| `outputs.lastJson(step)` reads that back without depending on block ids, which Deepnote reassigns | ||
| when it creates a notebook. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| // A pipeline in ~40 lines: fan out, gate on the results, then decide. | ||
| // | ||
| // Nothing here is browser- or Node-specific. The same code runs in a page — every step is an HTTP | ||
| // call to Deepnote, so there is no server, no local kernel, and no orchestration daemon. | ||
| // | ||
| // In a real project, import from '@deepnote/local-runner' after installing it. | ||
| import { orchestrate } from '../../../packages/local-runner/dist/index.js' | ||
|
|
||
| try { | ||
| process.loadEnvFile() | ||
| } catch {} | ||
|
|
||
| const token = process.env.DEEPNOTE_TOKEN | ||
| if (!token) { | ||
| console.error('Set DEEPNOTE_TOKEN (or put it in a .env) to run this example.') | ||
| process.exit(1) | ||
| } | ||
|
|
||
| // Notebooks already in Deepnote. Pass their ids in rather than editing this file. | ||
| const REGIONS = [ | ||
| { name: 'North America', notebookId: process.env.NA_NOTEBOOK_ID }, | ||
| { name: 'Europe', notebookId: process.env.EU_NOTEBOOK_ID }, | ||
| { name: 'Asia Pacific', notebookId: process.env.APAC_NOTEBOOK_ID }, | ||
| ].filter(region => region.notebookId) | ||
|
|
||
| if (REGIONS.length === 0) { | ||
| console.error('Set NA_NOTEBOOK_ID / EU_NOTEBOOK_ID / APAC_NOTEBOOK_ID to the notebooks to run.') | ||
| process.exit(1) | ||
| } | ||
|
|
||
| const QUALITY_THRESHOLD = 0.95 | ||
|
|
||
| const { value, graph, durationMs } = await orchestrate( | ||
| async ({ run, control, outputs }) => { | ||
| // Ordinary control flow is the pipeline: these fan out concurrently because Promise.all does. | ||
| const analyses = await Promise.all( | ||
| REGIONS.map(region => | ||
| run({ | ||
| id: `analyze-${slug(region.name)}`, | ||
| label: `${region.name} analysis`, | ||
| notebookId: region.notebookId, | ||
| inputs: { region: region.name, trailing_months: 6 }, | ||
| }) | ||
| ) | ||
| ) | ||
| const readings = analyses.map(step => outputs.lastJson(step)) | ||
|
|
||
| // A control node is a local decision that should still show up in the graph. | ||
| const belowThreshold = await control( | ||
| { | ||
| id: 'quality-gate', | ||
| kind: 'gate', | ||
| label: `${QUALITY_THRESHOLD * 100}% quality gate`, | ||
| dependsOn: analyses.map(step => step.id), | ||
| }, | ||
| () => readings.filter(reading => reading.qualityScore < QUALITY_THRESHOLD).map(reading => reading.region) | ||
| ) | ||
|
|
||
| return { checked: readings.length, belowThreshold } | ||
| }, | ||
| { token, onEvent: event => event.type === 'step_started' && console.log(` → ${event.stepId}`) } | ||
| ) | ||
|
|
||
| console.log(`\n ${value.checked} regions in ${(durationMs / 1000).toFixed(1)}s`) | ||
| console.log(` below threshold: ${value.belowThreshold.join(', ') || 'none'}`) | ||
| console.log(` graph: ${graph.nodes.length} nodes, ${graph.edges.length} edges\n`) | ||
|
|
||
| function slug(value) { | ||
| return value.toLowerCase().replace(/[^a-z0-9]+/g, '-') | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,124 @@ | ||
| import type { NormalizedRun, PollOptions, RunInputValue, WaitForRunSnapshotOptions } from '@deepnote/cloud' | ||
| import { | ||
| describeRunError, | ||
| isSuccessStatus, | ||
| pollRunUntilComplete, | ||
| triggerNotebookRun, | ||
| waitForRunSnapshot, | ||
| } from '@deepnote/cloud' | ||
| import { extractOutputs } from './extract-outputs' | ||
| import { finishResult, type OrchestrationStepExecutor } from './orchestrate' | ||
| import { parseSnapshot } from './snapshot-view' | ||
|
|
||
| /** | ||
| * The step executor every pipeline uses by default: three plain HTTP calls per step — start a run, | ||
| * poll it, read its snapshot. | ||
| * | ||
| * It is `fetch` and nothing else, which is what makes a pipeline portable. The same executor runs | ||
| * in a Node script, in CI, and in a browser page; there is no server in the middle and no local | ||
| * kernel involved. | ||
| */ | ||
|
|
||
| /** The API origin Deepnote Cloud serves from, used when a caller does not name one. */ | ||
| export const DEFAULT_CLOUD_API_URL = 'https://api.deepnote.com' | ||
|
|
||
| export interface CloudExecutorOptions { | ||
| /** | ||
| * Deepnote API token. | ||
| * | ||
| * In a published app this is the short-lived, viewer-scoped token the Deepnote shell issues, and | ||
| * it arrives with the origin to send it to. In a script it is an ordinary API token. | ||
| */ | ||
| token: string | ||
| /** API origin. Pair this with the token that was issued for it. */ | ||
| baseUrl?: string | ||
| /** Poll tuning shared by every step. */ | ||
| poll?: Omit<PollOptions, 'onStatus'> | ||
| /** Snapshot-settling tuning shared by every step. */ | ||
| snapshot?: WaitForRunSnapshotOptions | ||
| /** Abort every in-flight request for this pipeline. */ | ||
| signal?: AbortSignal | ||
| } | ||
|
|
||
| /** | ||
| * Coerce a pipeline's input value to what `POST /v2/runs` accepts. | ||
| * | ||
| * The API takes exactly `string | boolean | string[]`, so numbers are stringified here rather than | ||
| * rejected — a pipeline computing `monthly_target_k` should not have to remember to call | ||
| * `String()`. Anything without an unambiguous textual form is refused instead of guessed at. | ||
| */ | ||
| export function toRunInputs(inputs: Record<string, unknown>): Record<string, RunInputValue> { | ||
| const coerced: Record<string, RunInputValue> = {} | ||
| for (const [name, value] of Object.entries(inputs)) { | ||
| if (typeof value === 'string' || typeof value === 'boolean') { | ||
| coerced[name] = value | ||
| } else if (typeof value === 'number') { | ||
| if (!Number.isFinite(value)) { | ||
| throw new Error(`Input "${name}" is ${String(value)}, which Deepnote cannot accept.`) | ||
| } | ||
| coerced[name] = String(value) | ||
| } else if (Array.isArray(value) && value.every(part => typeof part === 'string')) { | ||
| coerced[name] = value as string[] | ||
| } else if (value !== undefined && value !== null) { | ||
| throw new Error( | ||
| `Input "${name}" is a ${Array.isArray(value) ? 'mixed array' : typeof value}. Deepnote inputs accept a string, boolean, or array of strings.` | ||
| ) | ||
| } | ||
| } | ||
| return coerced | ||
| } | ||
|
|
||
| /** Build the executor {@link orchestrate} uses. Exported for callers composing their own engine. */ | ||
| export function createCloudStepExecutor(options: CloudExecutorOptions): OrchestrationStepExecutor { | ||
| const baseUrl = options.baseUrl ?? DEFAULT_CLOUD_API_URL | ||
| const { token } = options | ||
|
|
||
| return async ({ id, step, startedMs, startedAt, emit }) => { | ||
| if (!step.notebookId) { | ||
| throw new Error(`Orchestration step "${id}" has no notebookId to run.`) | ||
| } | ||
|
|
||
| const started = await triggerNotebookRun( | ||
| baseUrl, | ||
| token, | ||
| { notebookId: step.notebookId, inputs: toRunInputs(step.inputs ?? {}) }, | ||
| { signal: options.signal } | ||
| ) | ||
| emit({ type: 'step_status', stepId: id, status: started.status }) | ||
|
|
||
| const completed: NormalizedRun = await pollRunUntilComplete(baseUrl, token, started.runId, { | ||
| ...options.poll, | ||
| signal: options.signal, | ||
| onStatus: status => { | ||
| emit({ type: 'step_status', stepId: id, status }) | ||
| }, | ||
| } as PollOptions) | ||
|
coderabbitai[bot] marked this conversation as resolved.
Resolved
|
||
|
|
||
| // Read the snapshot even when the run failed: it is usually the only place the failing block's | ||
| // error is recorded, and a page that shows nothing is worse than one that shows why. | ||
| const settled = await waitForRunSnapshot(baseUrl, token, completed, { | ||
| ...options.snapshot, | ||
| signal: options.signal, | ||
| }) | ||
| const snapshotYaml = settled.content | ||
| const success = isSuccessStatus(completed.status) | ||
|
|
||
| return finishResult( | ||
| { | ||
| id, | ||
| target: 'cloud', | ||
| success, | ||
| status: completed.status, | ||
| outputs: snapshotYaml ? extractOutputs(snapshotYaml) : [], | ||
| snapshotYaml, | ||
| snapshot: snapshotYaml ? parseSnapshot(snapshotYaml) : null, | ||
| runId: completed.runId, | ||
| error: success | ||
| ? undefined | ||
| : (describeRunError(completed) ?? `the run finished with status "${completed.status}"`), | ||
|
coderabbitai[bot] marked this conversation as resolved.
Resolved
|
||
| }, | ||
| startedMs, | ||
| startedAt | ||
| ) | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| import type { RunBlockOutput } from './run-with-inputs' | ||
| import type { SnapshotView } from './snapshot-view' | ||
| import { parseSnapshot } from './snapshot-view' | ||
|
|
||
| /** | ||
| * Read the per-block outputs out of a cloud snapshot's YAML, in document order. | ||
| * | ||
| * Split out of `cloud-common.ts` so it can bundle for the browser: everything else there resolves | ||
| * tokens from `process.env`, which a page has no business carrying. Only `parseSnapshot` and the | ||
| * block schemas are reachable from here. | ||
| * | ||
| * Any executable block type carries outputs — code, SQL, visualization, big-number — so read them | ||
| * off whatever block has them rather than special-casing `code`. | ||
| * | ||
| * A snapshot that won't parse throws rather than returning nothing: the run succeeded, so "no | ||
| * outputs" is a claim about the notebook, and it would be a false one. The caller still has the raw | ||
| * YAML to inspect. | ||
| */ | ||
| export function extractOutputs(snapshotYaml: string): RunBlockOutput[] { | ||
| let view: SnapshotView | ||
| try { | ||
| view = parseSnapshot(snapshotYaml) | ||
| } catch (error) { | ||
| throw new Error( | ||
| `Deepnote returned a snapshot that could not be parsed: ${error instanceof Error ? error.message : String(error)}` | ||
| ) | ||
| } | ||
| const outputs: RunBlockOutput[] = [] | ||
| for (const notebook of view.notebooks) { | ||
| for (const block of notebook.blocks) { | ||
| if (block.outputs.length > 0) { | ||
| outputs.push({ blockId: block.id, outputs: block.outputs, executionCount: block.executionCount }) | ||
| } | ||
| } | ||
| } | ||
| return outputs | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.