Skip to content

Commit 928168d

Browse files
jankoritakjankoritak
authored
Merge branch 'main' into feat/cli-rename-notebook
2 parents e7ece48 + 8c0b8c8 commit 928168d

80 files changed

Lines changed: 6536 additions & 379 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/ci.yml‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -226,6 +226,44 @@ jobs:
226226
- name: Test CLI --help
227227
run: node dist/bin.js --help
228228
working-directory: packages/cli
229+
runtime-integration:
230+
name: Runtime Integration (deepnote-toolkit)
231+
runs-on: ubuntu-latest
232+
timeout-minutes: 20
233+
steps:
234+
- name: Checkout code
235+
uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
236+
237+
- name: Setup pnpm
238+
uses: pnpm/action-setup@b906affcce14559ad1aafd4ab0e942779e9f58b1 # v4
239+
240+
- name: Setup Node.js
241+
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
242+
with:
243+
node-version-file: '.nvmrc'
244+
cache: 'pnpm'
245+
246+
- name: Setup Python
247+
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
248+
with:
249+
python-version: '3.12'
250+
251+
- name: Install deepnote-toolkit with its server extra
252+
run: |
253+
python -m pip install --upgrade pip
254+
python -m pip install "deepnote-toolkit[server]"
255+
256+
- name: Install dependencies
257+
run: pnpm install --frozen-lockfile
258+
259+
- name: Build packages
260+
run: pnpm run build
261+
262+
# Runs the CLI and the execution engine against the real toolkit server and kernel.
263+
- name: Run integration tests
264+
run: pnpm run test:integration
265+
env:
266+
DEEPNOTE_PYTHON: ${{ env.pythonLocation }}/bin/python
229267
license-check:
230268
name: License Check
231269
runs-on: ubuntu-latest

‎AGENTS.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,10 @@ pnpm test
5656

5757
# Run tests with coverage
5858
pnpm test:coverage
59+
60+
# Run the integration tests against a real deepnote-toolkit server (excluded from `pnpm test`).
61+
# Needs `pip install "deepnote-toolkit[server]"`; point DEEPNOTE_PYTHON at that interpreter.
62+
pnpm test:integration
5963
```
6064

6165
#### Type Checking
@@ -119,6 +123,7 @@ Always run these checks before considering work complete:
119123
- Test edge cases, error handling, and special characters
120124
- For functions that generate code, test the exact output format
121125
- Tests must not depend on live network calls or real Deepnote Cloud credentials — mock external APIs. Verifying behavior against the real Deepnote Cloud API is a manual, explicitly-requested step outside `pnpm test`, and any resources created that way (projects, notebooks, runs) must be cleaned up afterward
126+
- Tests that start the real `deepnote-toolkit` server belong in `*.integration.test.ts` files. They are excluded from `pnpm test`, run with `pnpm test:integration`, and are exercised in CI by the "Runtime Integration" job. Run only one integration suite at a time per machine: its leaked-process guard sees every toolkit process of the interpreter, so a concurrent run's servers are reported as leaks
122127

123128
#### TypeScript Guidelines
124129

@@ -211,6 +216,17 @@ const testFixturesDir = path.join(__dirname, "../../../test-fixtures");
211216
const fixturePath = path.join(testFixturesDir, "my-fixture.ipynb");
212217
```
213218

219+
## Keeping Documentation in Sync
220+
221+
Before opening a pull request, check whether the change makes any documentation stale, and update it in the same pull request:
222+
223+
- `docs/` — user-facing product documentation, published at <https://deepnote.com/docs>. This directory is the source of truth for the whole product, including features developed in other repositories. Update it whenever user-visible behavior changes: CLI commands and flags, the `.deepnote` format, hosted MCP, integrations, or app behavior.
224+
- `packages/<name>/README.md` and `packages/<name>/docs/` — reference for the package you changed.
225+
- `skills/deepnote/references/` — agent-facing references; see the rules below.
226+
- `README.md`, `CONTRIBUTING.md`, and `FILES.md` — repository layout, setup, and workflow changes.
227+
228+
Update only the pages your change actually affects. An unrelated documentation rewrite belongs in its own pull request.
229+
214230
## Keeping the Deepnote Skill in Sync
215231

216232
The `skills/deepnote/` directory contains reference documentation used by AI agents. When making changes to any of the following, you **must** also update the corresponding skill files:

‎CONTRIBUTING.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,14 @@ Run tests with coverage:
8282
pnpm test:coverage
8383
```
8484

85+
Run the integration tests, which start a real `deepnote-toolkit` server (they are excluded from
86+
`pnpm test`; install the toolkit with `pip install "deepnote-toolkit[server]"` and point
87+
`DEEPNOTE_PYTHON` at that interpreter):
88+
89+
```bash
90+
pnpm test:integration
91+
```
92+
8593
Run tests in watch mode (in a specific package):
8694

8795
```bash

‎cspell.json‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@
3030
"words": [
3131
"agentic",
3232
"asname",
33+
"autorestarting",
3334
"awsathena",
3435
"bijective",
3536
"Braund",
@@ -64,13 +65,15 @@
6465
"extglob",
6566
"fflate",
6667
"frontends",
68+
"getppid",
6769
"ggplot",
6870
"github",
6971
"graphviz",
7072
"groupby",
7173
"hotreload",
7274
"Instantiator",
7375
"iopub",
76+
"ipykernel",
7477
"ipynb",
7578
"ipywidgets",
7679
"isnull",
@@ -84,6 +87,7 @@
8487
"kimi",
8588
"kiro",
8689
"kpis",
90+
"lumino",
8791
"macchiato",
8892
"millis",
8993
"mindsdb",
@@ -101,10 +105,13 @@
101105
"oss",
102106
"Papermill",
103107
"Pclass",
108+
"pgrep",
104109
"pgsql",
110+
"pids",
105111
"plannable",
106112
"pnpm",
107113
"pyformat",
114+
"pylsp",
108115
"pymongo",
109116
"pymssql",
110117
"pypi",

‎docs/custom-environment.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ Custom environments are especially valuable for complex projects where dependenc
2020

2121
We recommend building your custom Dockerfile on top of our `deepnote/python:3.x` base image to ensure compatibility with all Deepnote features. If you're creating an environment from scratch, your image must meet these requirements:
2222

23-
- Python (versions 3.10-3.13) installed and accessible via the `python` command
23+
- Python (versions 3.10-3.14) installed and accessible via the `python` command
2424
- Built for the `linux/amd64` platform (M1 Mac users: use `-platform linux/amd64` flag)
2525
- Functioning `pip` installation that can install packages to Python's path
2626
- `bash` and `curl` installed

‎docs/deepnote-cli-publish.md‎

Lines changed: 19 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -32,9 +32,9 @@ access model differs from [data apps](/docs/data-apps), which do offer public an
3232

3333
## Authentication
3434

35-
The CLI reads your token from the `DEEPNOTE_TOKEN` environment variable, or from an explicit
36-
`--token` flag. Create a token in your workspace under
37-
[Settings & members → API tokens](https://deepnote.com/workspace/settings/api-tokens).
35+
The CLI reads your token from the `DEEPNOTE_TOKEN` environment variable, from a `.env` file in the
36+
current directory, or from an explicit `--token` flag. Create an API key in your workspace under
37+
**Settings & members → Security → API keys** (see the [Deepnote API docs](/docs/deepnote-api)).
3838

3939
```bash
4040
export DEEPNOTE_TOKEN="<your-token>"
@@ -55,8 +55,10 @@ An API token carries your access to the workspace. Treat it like a password.
5555
`DEEPNOTE_TOKEN` for the publish step only. Never commit it to the repository you are deploying.
5656
- **Rotate and revoke** from the same settings page if a token is ever exposed.
5757
- **Keep it out of the build directory.** Everything under the directory you publish becomes readable
58-
at the site URL by anyone who can view the site — including dotfiles, source maps, and stray `.env`
59-
files. Publish a clean build output directory, not a project root.
58+
at the site URL by anyone who can view the site — including dotfiles and source maps. Publish a
59+
clean build output directory, not a project root. As a safeguard, the CLI refuses to publish a
60+
directory that contains a `.env` or `.env.*` file anywhere inside it (exit code `2`, nothing is
61+
uploaded).
6062

6163
## Finding a project ID
6264

@@ -136,10 +138,18 @@ data app is the model that supports it — not a published static site.
136138
By default a published site is a plain static website: it can serve HTML, CSS, JavaScript, and
137139
assets, but it cannot call the Deepnote API.
138140

139-
Passing `--api-access enabled` lets the page acquire a short-lived, project- and viewer-scoped token
140-
from the Deepnote shell that embeds it. That token has a deliberately narrow surface — read the
141-
configured notebook, start a run, poll that run — which is what makes an interactive page possible
142-
without a server of your own.
141+
Passing `--api-access enabled` lets the page acquire a project- and viewer-scoped token from the
142+
Deepnote shell that embeds it. The token expires 15 minutes after it is minted and has a deliberately
143+
narrow surface — read the configured notebook's inputs and block metadata (no block source), start a
144+
detached run, and poll that run for its outputs as `snapshotBlocks` — which is what makes an
145+
interactive page possible without a server of your own. Every other endpoint answers 403, so a
146+
feature that works in a local preview with a personal token can break only once embedded.
147+
148+
The page obtains the token by posting a `deepnote-static-files-api-token-request` message to the
149+
shell origin, which replies with the token, the API origin to send it to, and its expiry. Expiry is
150+
not a permanent failure: repeat that request to receive a fresh token, ideally shortly before the
151+
current one expires and again on a 401. `examples/local-runner/cloud-app` implements the handshake
152+
and the refresh.
143153

144154
This is a second opt-in layered on top of site sharing, and it can only ever narrow the audience, not
145155
widen it: a viewer who cannot see the site cannot obtain a token for it. Because every viewer is a

‎docs/deepnote-cli-sync.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -34,9 +34,9 @@ They can be used together, but they are separate mechanisms with separate state.
3434

3535
## Authentication
3636

37-
The CLI reads your token from the `DEEPNOTE_TOKEN` environment variable, or from an explicit
38-
`--token` flag. Create a token in your workspace under
39-
[Settings & members → API tokens](https://deepnote.com/workspace/settings/api-tokens).
37+
The CLI reads your token from the `DEEPNOTE_TOKEN` environment variable, from a `.env` file in the
38+
sync directory, or from an explicit `--token` flag. Create an API key in your workspace under
39+
**Settings & members → Security → API keys** (see the [Deepnote API docs](/docs/deepnote-api)).
4040

4141
```bash
4242
export DEEPNOTE_TOKEN="<your-token>"

‎package.json‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@
2929
"spell-check:fix": "cspell \"**/*.{md,txt,json,js,ts,tsx,jsx,py,yml,yaml}\" --no-exit-code --show-suggestions",
3030
"test": "vitest run",
3131
"test:coverage": "vitest run --coverage",
32+
"test:integration": "vitest run --config vitest.integration.config.ts",
3233
"typecheck": "tsc --noEmit -p tsconfig.json && pnpm -r exec tsc --noEmit"
3334
},
3435
"lint-staged": {
@@ -88,7 +89,7 @@
8889
"flatted": ">=3.4.2",
8990
"glob": ">=11.1.0",
9091
"hono": ">=4.13.5",
91-
"ip-address": ">=10.3.1",
92+
"ip-address": ">=10.5.1",
9293
"js-yaml": ">=4.3.2 <5",
9394
"lodash": ">=4.18.0",
9495
"lodash-es": ">=4.18.0",
@@ -100,7 +101,7 @@
100101
"qs": ">=6.16.0",
101102
"rollup": ">=4.59.0",
102103
"smol-toml": ">=1.6.1",
103-
"undici": "6.28.0",
104+
"undici": "6.28.1",
104105
"vite": ">=8.0.16",
105106
"vitest": "4.1.11",
106107
"ws": ">=8.20.1",

‎packages/blocks/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ This package provides TypeScript types and utilities for working with Deepnote n
3131
### Display Blocks
3232

3333
- **visualization**: Interactive charts using Vega-Lite
34+
- **pivot-table**: Code-free cross-tabulation of a DataFrame (beta)
3435
- **big-number**: KPI display with Jinja2 templates
3536
- **button**: Interactive button with variable control
3637

‎packages/blocks/src/blocks/code-blocks.ts‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,14 @@ import type { CodeBlock, DeepnoteBlock } from '../deepnote-file/deepnote-file-sc
44
import { createDataFrameConfig } from './data-frame'
55

66
export function createPythonCodeForCodeBlock(block: CodeBlock): string {
7+
// IPython only recognizes a cell magic (`%%bash`, `%%time`, ...) when it is on the
8+
// first non-blank line of the cell, at column zero. Prepending the DataFrame config
9+
// would push it down and make IPython parse the cell as Python, so emit the content
10+
// as-is. The config would be meaningless there anyway: the cell body is not Python.
11+
if (startsWithCellMagic(block.content)) {
12+
return block.content
13+
}
14+
715
const dataFrameConfig = createDataFrameConfig(block)
816

917
return dedent`
@@ -16,3 +24,11 @@ export function createPythonCodeForCodeBlock(block: CodeBlock): string {
1624
export function isCodeBlock(block: DeepnoteBlock): block is CodeBlock {
1725
return block.type === 'code'
1826
}
27+
28+
function startsWithCellMagic(content: string | undefined): content is string {
29+
// Mirrors IPython's input cleanup: leading blank lines are dropped, but indentation
30+
// before `%%` is not reliably stripped, so it must sit at column zero.
31+
const firstNonBlankLine = content?.split('\n').find(line => line.trim() !== '')
32+
33+
return firstNonBlankLine?.startsWith('%%') ?? false
34+
}

0 commit comments

Comments
 (0)