Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions examples/local-runner/scheduled-pipeline/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# scheduled-pipeline

Run a `.deepnote` pipeline **on a schedule**, with no workflow engine and no server of your own.

Deepnote already schedules one notebook. Point that at [`runner.deepnote`](./runner.deepnote) and a
whole pipeline runs on a schedule: the run is an ordinary Deepnote run — durable, retryable, and
visible in Deepnote's own UI — while the pipeline definition stays in a file.

## Why not just schedule the pipeline file itself?

Because Deepnote's block engine would do the wrong thing quietly, which is worse than failing.

- It **runs blocks strictly in order**, so a fan-out that should be concurrent is serialized.
- It **knows nothing about `run_if`, `for_each`, or `{{ }}`** — those are read by this interpreter,
not by Deepnote. A conditional recovery step would run unconditionally, a fan-out would run once,
and `{{portfolio}}` would be passed through as that literal string.
- Native `notebook-function` inputs are **baked as a static JSON literal** at authoring time, so no
value can flow from one step into the next.

So the scheduled artifact is a _runner_ that interprets the manifest, not the manifest itself.

## Use it

1. Create `runner.deepnote` as a notebook in your Deepnote project.
2. Put your pipeline manifest in the same project (see
[`../sales-pipeline.deepnote`](../sales-pipeline.deepnote)) and set the **manifest path** input.
3. Set `DEEPNOTE_TOKEN` in the project's environment variables.
4. Schedule the notebook.

The notebook is self-contained: the interpreter is embedded in a code block, so it does not depend
on this repository being present. Its CLI entry point is stripped, because a notebook cell runs with
`__name__ == "__main__"` and would otherwise try to parse command-line arguments.

## Run it locally first

```bash
python3 packages/local-runner/python/deepnote_pipeline.py --plan examples/local-runner/sales-pipeline.deepnote
DEEPNOTE_TOKEN=… python3 packages/local-runner/python/deepnote_pipeline.py --run examples/local-runner/sales-pipeline.deepnote
```

`--plan` prints the DAG and runs nothing, which is the fastest way to check a manifest.

## Two implementations, one set of semantics

The interpreter exists in TypeScript (for the browser and scripts) and in Python (here). Two
implementations of one language is a standing risk that they quietly diverge, so
[`test-fixtures/pipeline-conformance`](../../../test-fixtures/pipeline-conformance) is the contract:
both planners must produce identical plans for every fixture, and `pipeline-conformance.test.ts`
fails if they do not.

If you change `run_if`, `for_each`, `{{ }}`, or dependency derivation, change it in both and add a
fixture that would have caught the difference.

## What this does not do

Steps run concurrently within the notebook run, and the run itself is durable — but there is no
resume: if the notebook run fails halfway, rerunning starts from the beginning. Notebook runs are
not automatically idempotent, so re-running a pipeline re-runs its side effects. For replay,
per-step retries, and timers, use the Workflow SDK integration in
[`@deepnote/local-runner/workflows`](../../../packages/local-runner/README.md) instead.
803 changes: 803 additions & 0 deletions examples/local-runner/scheduled-pipeline/runner.deepnote

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
"example:gallery": "pnpm --filter @deepnote/local-runner... build && node examples/local-runner/gallery/serve.mjs",
"example:local-runner": "pnpm --filter @deepnote/local-runner... build && node examples/local-runner/run-app/serve.mjs",
"example:orchestration": "pnpm --filter @deepnote/local-runner... build && node examples/local-runner/orchestration/run.mjs",
"example:pipeline-plan": "python3 packages/local-runner/python/deepnote_pipeline.py --plan examples/local-runner/sales-pipeline.deepnote",
"example:schedule-cloud": "pnpm --filter @deepnote/local-runner... build && node examples/local-runner/schedule-cloud.mjs",
"example:snapshot-viewer": "pnpm --filter @deepnote/local-runner... build && node examples/local-runner/snapshot-viewer/serve.mjs",
"license-check": "license-checker-rseidelsohn --json --onlyAllow \"MIT;Apache-2.0;BSD-2-Clause;BSD-3-Clause;ISC\" --excludePackages \"deepnote\"",
Expand Down
25 changes: 25 additions & 0 deletions packages/local-runner/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -369,6 +369,31 @@ boundary worth keeping.

See [`examples/local-runner/sales-pipeline.deepnote`](../../examples/local-runner/sales-pipeline.deepnote).

### Run a pipeline on a schedule

Deepnote already schedules one notebook. Point that at a notebook that _interprets_ a manifest and a
whole pipeline runs on a schedule — the run is an ordinary Deepnote run, durable and visible in
Deepnote's UI, with no workflow engine and no server of your own.

```bash
python3 packages/local-runner/python/deepnote_pipeline.py --plan pipeline.deepnote # print the DAG
DEEPNOTE_TOKEN=… python3 packages/local-runner/python/deepnote_pipeline.py --run pipeline.deepnote
```

Scheduling the _manifest_ itself does not work, and fails quietly rather than loudly: Deepnote's
block engine runs blocks in order (serializing the fan-out), knows nothing about `run_if`,
`for_each`, or `{{ }}`, and bakes `notebook-function` inputs as a static literal so no value can flow
between steps. The scheduled artifact has to be a runner, not the definition.

See [`examples/local-runner/scheduled-pipeline`](../../examples/local-runner/scheduled-pipeline) for
a self-contained notebook you can create once and schedule.

**Two implementations, one contract.** The interpreter exists in TypeScript for the browser and in
Python for a scheduled notebook, which is a standing risk of quiet divergence.
`test-fixtures/pipeline-conformance` is the contract: both planners must produce identical plans for
every fixture, enforced by `pipeline-conformance.test.ts`. Change a semantic in one language and that
test fails until it is changed in the other.

### Make a pipeline durable

`orchestrate` holds its state in one process and is gone if that process is. That is the right trade
Expand Down
7 changes: 4 additions & 3 deletions packages/local-runner/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -33,10 +33,11 @@
"module": "./dist/index.js",
"types": "./dist/index.d.ts",
"files": [
"dist",
"README.md",
"LICENSE",
"package.json"
"README.md",
"dist",
"package.json",
"python"
],
"scripts": {
"build": "tsdown",
Expand Down
Loading
Loading