This guide is for developers contributing to the StatBus codebase.

For deploying StatBus, see Deployment Guide.
For using StatBus, see User Guide.

• Development Setup
• Project Structure
• Local Development Workflow
• Database Development
• Frontend Development
• Code Conventions
• Testing
• Architecture

Required Tools:

• Docker 24.0+ and Docker Compose 2.20+
• Git 2.40+
• Node.js (version specified in .nvmrc)
• pnpm 8.0+

Platform-Specific:

macOS:

# Install Homebrew
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

# Install tools
brew install nvm git docker docker-compose crystal-lang
brew install --cask docker  # Docker Desktop

Linux (Ubuntu/Debian):

# Install Docker
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

# Install Crystal (for database migrations)
curl -fsSL https://crystal-lang.org/install.sh | sudo bash

# Verify Crystal installation
crystal --version
shards --version

# Install Node Version Manager
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash

# Install pnpm
curl -fsSL https://get.pnpm.io/install.sh | sh -

Windows:

# Install Scoop
Set-ExecutionPolicy RemoteSigned -Scope CurrentUser
irm get.scoop.sh | iex

# Install tools
scoop install git nvm
scoop bucket add extras
scoop install docker

git clone https://github.com/statisticsnorway/statbus.git
cd statbus

Set hooks path (enforces conventions):

git config core.hooksPath .githooks

Configure line endings (critical for cross-platform):

git config --global core.autocrlf true

This project uses LF line endings. Git on Windows may convert to CRLF, which breaks scripts.

Build the StatBus CLI tool for database migrations:

cd cli && go build -o ../sb .

This compiles the Go CLI tool to ./sb.

cp .users.example .users.yml

Edit .users.yml to add your development users:

users:
  - email: dev@example.com
    password: devpassword
    role: admin_user

./sb config generate

This creates .env, .env.credentials, and .env.config with development defaults.

# Use Node version from .nvmrc
cd app
nvm install
nvm use

# Install pnpm globally
npm install -g pnpm

# Install app dependencies
pnpm install

statbus/
├── app/                    # Next.js frontend application
│   ├── src/               # Source code
│   ├── public/            # Static assets
│   ├── package.json       # Node dependencies
│   └── CONVENTIONS.md     # Frontend conventions
├── cli/                   # Crystal CLI tool
│   ├── src/              # CLI source code
│   └── bin/              # Compiled binaries
├── ops/                  # Operations scripts
│   └── maintenance/      # Maintenance page
├── .githooks/            # Git hooks
├── doc/                  # Documentation
│   ├── integration/      # API & PostgreSQL guides
│   ├── deployment/       # Deployment guides
│   └── service-architecture.md
├── migrations/           # Database migrations
├── test/                # Database tests
│   ├── sql/            # Test SQL files
│   └── expected/       # Expected output
├── postgres/           # PostgreSQL configuration
├── caddy/             # Caddy configuration
├── rest/              # PostgREST configuration
├── worker/            # Background worker (Crystal)
├── .env.config        # Deployment configuration (edit this)
├── .env.credentials   # Generated credentials (don't edit)
├── .env               # Generated environment (don't edit)
├── CONVENTIONS.md     # Backend coding conventions
├── AGENTS.md          # AI agent guide
└── README.md          # Gateway document

In development mode, you run backend services in Docker and the Next.js app locally for hot-reload.

# Start PostgreSQL, PostgREST, Caddy, Worker
./sb start all_except_app

# Initialize database (first time only)
./dev.sh create-db
./sb users create
./sb migrate up

cd app
nvm use
pnpm install
pnpm run dev

Access the application:

• Frontend: http://localhost:3000
• API via Caddy: http://localhost:3010/rest/
• Supabase Studio: http://localhost:3001

./sb stop all

The development environment provides PostgreSQL access through Caddy's Layer4 TLS proxy.

Quick Access:

# Use helper script (recommended)
./sb psql

# Or connect manually
eval $(./sb config show --postgres)
psql

Manual Connection:

export PGHOST=local.statbus.org
export PGPORT=3024
export PGDATABASE=statbus_speed
export PGUSER=postgres
export PGPASSWORD=$(./sb dotenv -f .env get POSTGRES_ADMIN_PASSWORD)
export PGSSLNEGOTIATION=direct
export PGSSLMODE=require
export PGSSLSNI=1

psql

Connection Details:

• Domain: local.statbus.org (resolves to 127.0.0.1)
• Port: 3024 (from CADDY_DB_PORT)
• Database: statbus_speed (from POSTGRES_APP_DB)
• TLS: Self-signed internal CA
• SSL Mode: require (encrypted, no cert verification)

StatBus uses a versioned migration system with Crystal CLI.

Create Migration:

cd cli
./bin/statbus migrate new --description "add column to legal_unit"

This creates two files in migrations/:

• YYYYMMDDHHmmSS_add_column_to_legal_unit.up.sql
• YYYYMMDDHHmmSS_add_column_to_legal_unit.down.sql

Apply Migrations:

./cli/bin/statbus migrate up      # Apply all pending
./cli/bin/statbus migrate down    # Rollback last migration
./cli/bin/statbus migrate redo    # Rollback and re-apply last

Migration Conventions:

• Write idempotent migrations (can run multiple times)
• Always provide down migration for rollback
• Test migrations on sample data before committing
• See CONVENTIONS.md for SQL style guide

A seed is a pg_dump of a fully-migrated, empty database. A fresh install or dev box restores it in ~2 seconds instead of replaying every migration from zero, then runs only the migrations newer than the seed. Creating the seed is a developer/CI activity (documented here); restoring it is part of install — see Upgrade Timeline § Fresh-install seed restore for where the restore enters the runtime.

./sb db seed dump (cli/cmd/seed.go:285, core in DumpSeed, cli/cmd/seed.go:379) writes two files into .db-seed/ from the statbus_seed database:

• .db-seed/seed.pg_dump — pg_dump -Fc --no-owner --exclude-table-data=auth.secrets <seed-db> (cli/cmd/seed.go:463). Custom format (-Fc) so the consume side can pg_restore. auth.secrets data is excluded because it holds per-deployment JWT secrets that must never ship in a shared artifact (cli/cmd/seed.go:447).
• .db-seed/seed.json — the metadata sidecar (cli/cmd/seed.go:490).

DumpSeed dumps from ${POSTGRES_SEED_DB} (the canonical fresh-from-migrations DB), never from the runtime app DB, which is contaminable by definition (cli/cmd/seed.go:376). It refuses to run if the DB is unreachable (cli/cmd/seed.go:384) or the seed DB does not exist (cli/cmd/seed.go:398), pointing you at ./dev.sh recreate-seed.

Build the seed DB itself with ./dev.sh recreate-seed (dev.sh:1326): it fetches the latest published seed, restores it, then applies only the newer migrations — the same fast path the install uses. Under the hood that is the three primitives ./sb db seed create-db (a fresh copy of template_statbus plus the per-DB auth schema and grants — CreateSeedDb, cli/cmd/seed.go:313) → ./sb migrate up --target seed → ./sb db seed dump. FULL_REPLAY=1 ./dev.sh recreate-seed forces a from-zero rebuild.

seedMeta (cli/cmd/seed.go:37) — written at creation, read on the consume side:

These fields are consumed on the restore side — see Upgrade Timeline § Fresh-install seed restore.

migrations/post_restore.sql is re-run on every migrate up, even when no new migrations are pending (cli/internal/migrate/migrate.go:756, applied at :912). So an edit to post_restore.sql changes the post-restore schema state without bumping any migration version — and without a fingerprint that change would silently ship a stale dump. Recording the file's sha256 in seed.json (cli/cmd/seed.go:437) makes the change detectable even when migration_version is unchanged; an absent field is treated as "fingerprint missing → full rebuild" (cli/cmd/seed.go:28).

The seed ships as the OCI image ghcr.io/statisticsnorway/statbus-seed:<commit_short> — the same commit-addressable transport as the five service images, replacing the former db-seed git branch. The seed job in .github/workflows/images.yaml:129 builds it --target seed from postgres/Dockerfile after the statbus-sb manifest exists (it is pulled in as a build-context), passing the full COMMIT SHA so seed.json carries the right commit when the build tree has no .git. It is amd64-only because the -Fc logical dump restores onto either architecture.

Inside the image build, the hermetic seed-builder stage (postgres/Dockerfile:452) runs the real sb subcommands — sb db seed create-db → sb migrate up --target seed → sb db seed dump --commit $COMMIT (postgres/Dockerfile:513) — so there is zero hand-mirrored SQL. The final seed stage is busybox:musl and ships both /seed.pg_dump and /seed.json with a self-documenting CMD (postgres/Dockerfile:534); docker run on it prints extraction usage rather than starting a service.

StatBus uses several PostgreSQL schemas:

• public: Main application tables (legal_unit, establishment, etc.)
• admin: Administrative tables (users, settings)
• auth: Authentication functions and JWT handling
• db: Views and helper functions
• lifecycle_callbacks: Triggers and validation logic

Run pg_regress tests (or ./dev.sh continous-integration-test for the local equivalent of GitHub Fast Tests):

# Run all tests
./dev.sh test all

# Run specific test
./dev.sh test 015_my_test

# Run failed tests only
./dev.sh test failed

Test files location:

• SQL: test/sql/*.sql
• Expected output: test/expected/*.out

Create a new test:

# Create SQL test file
echo "SELECT 'test output';" > test/sql/999_my_test.sql

# Run it to generate expected output
./dev.sh test 999_my_test

# If correct, copy output
cp test/regression.out test/expected/999_my_test.out

After changing database schema, regenerate TypeScript types:

./sb types generate

This updates app/src/lib/database.types.ts with current schema.

Run SQL file:

./sb psql < my_script.sql

Run SQL command:

echo "SELECT * FROM auth.users LIMIT 5;" | ./sb psql

Interactive psql:

./sb psql

The frontend is built with:

• Next.js 15 (App Router)
• React 18 with TypeScript
• Tailwind CSS for styling
• shadcn/ui component library
• Jotai for state management

cd app
pnpm run dev        # Start dev server with Turbopack

The dev server runs on http://localhost:3000 with:

• Hot module replacement
• TypeScript checking
• Fast refresh

cd app

# Development
pnpm run dev           # Start dev server
pnpm run build         # Production build
pnpm run start         # Start production server

# Code Quality
pnpm run lint          # ESLint
pnpm run format        # Check Prettier
pnpm run format:fix    # Fix Prettier issues
pnpm run tsc           # Type check

# Testing
pnpm run test          # Run Jest tests
pnpm run test:watch    # Watch mode

Critical Rules:

• Small, independent atoms prevent re-render loops
• If state can change independently, it MUST be in its own atom
• Use atomEffect for set-if-null patterns, NOT useEffect
• Variables match atom names: const timeContext = useAtomValue(timeContextAtom)

Use Guarded Effects:

import { useGuardedEffect } from '@/lib/use-guarded-effect';

// ALL effects MUST use useGuardedEffect
useGuardedEffect(callback, deps, 'FileName.tsx:purpose');

See app/CONVENTIONS.md for detailed frontend conventions.

Development Mode (pnpm run dev on host):

1. Browser accesses http://localhost:3000
2. Client makes API calls to /rest/*
3. Next.js dev server proxies to Caddy (http://localhost:3010/rest/*)
4. Caddy handles auth cookie conversion
5. Caddy proxies to PostgREST

Production Mode (Docker):

1. Browser accesses https://statbus.example.com
2. Client makes API calls directly to Caddy /rest/*
3. Caddy handles auth and proxies to PostgREST

See CONVENTIONS.md for full details.

Key patterns:

Function Definitions:

CREATE FUNCTION auth.jwt_verify(token_value text)
RETURNS auth.jwt_verify_result
LANGUAGE plpgsql
SECURITY DEFINER
SET search_path = public, auth, pg_temp
AS $jwt_verify$
DECLARE
  _jwt_verify_result auth.jwt_verify_result;
BEGIN
  -- Function body
END;
$jwt_verify$;

Naming Conventions:

• x_id = foreign key to table x
• x_ident = external identifier (not from DB)
• x_at = TIMESTAMPTZ
• x_on = DATE

Temporal Logic:

-- Chronological order: start <= point AND point < end
WHERE valid_from <= current_date AND current_date < valid_to

See app/CONVENTIONS.md for full details.

Import Style:

import { NextRequest, NextResponse } from "next/server";
import { getServerRestClient } from "@/context/RestClientStore";

Named Exports (preferred over default exports):

export const MyComponent = () => { ... };
export function myFunction() { ... }

Format: prefix: description

Prefixes:

• feat: - New feature
• fix: - Bug fix
• docs: - Documentation
• refactor: - Code refactoring
• test: - Test changes
• chore: - Build/tooling changes

Examples:

feat: Add temporal foreign key validation
fix: JWT verification with expired tokens
docs: Update PostgreSQL connection guide

# Run all tests
./dev.sh test all

# Run specific test
./dev.sh test 015_jwt_auth

# Run multiple tests
./dev.sh test 015_jwt_auth 020_temporal

# Exclude tests
./dev.sh test all -010_old_test

# Run failed tests
./dev.sh test failed

cd app
pnpm run test           # Run once
pnpm run test:watch     # Watch mode
pnpm run test:coverage  # With coverage

Between the unit tests and the paid VM harness is a Go build-tag tier that drives real upgrade and install code against the local Docker Compose database. Fast Tests runs this tier on every commit after the daemon-floor oracle.

# Keep Compose running and migrate statbus_seed first (for example,
# ./dev.sh migrate-and-test fast). No app database is modified.
./dev.sh test-livedb

# The stable-release gates need a real RC, credentials, and canary access, so
# they remain manual under their separate mechanism tag.
STATBUS_LIVE_RELEASE_GATES=v2026.09.0-rc.12 GITHUB_TOKEN=$(gh auth token) \
  go test -C cli -tags release_live -count=1 -run TestReleaseStablePreflightGates -v ./cmd/release

Each package creates one detached Git worktree at HEAD, copies the local .env.config and .env.credentials, generates that worktree's .env, and builds an identity-bearing sb with the same ldflags as dev.sh. Each package clones the migrated statbus_seed into its own disposable database, points only its copied .env at that clone, and drops it after the package completes. This avoids both CI's uninitialized app database and local upgrade rows. Direct subprocess calls use a pinned copy for the whole package run. Mutable daemon files such as .env, sb, sb.old, tmp/, and migration fixtures therefore belong to the scratch worktree, never the developer's checkout. The install and upgrade packages share a filesystem flock because they also exercise host-level maintenance and upgrade files.

Rules they follow, and that any new one must follow:

• Select the tier with //go:build livedb, not an environment-variable skip.
• Name each test for the claim its assertions prove. Put ticket provenance in a comment rather than the test name.
• Refuse to run if a scratch-worktree upgrade marker, maintenance flag, or sb.old exists.
• Answer any docker compose call with a PATH shim. Tests use the host's Compose database, but never mutate or bind-mount the main checkout into a container.
• Leave the read-only default OFF and database rows as found.

To verify the upgrade service's rollback and recovery mechanisms, use these procedures. Each requires tagging deliberate broken RCs, then fixing them.

Test 1: Broken migration → database rollback

1. Create a migration that fails (e.g., SELECT 1/0;)
2. Optionally add a preceding migration that creates a marker table (to verify both are rolled back)
3. Tag as a pre-release RC
4. Schedule on a test server: ./sb upgrade register <broken-rc> && ./sb upgrade schedule <broken-rc>
5. Verify: migration fails, service rolls back, marker table is gone, upgrade shows "rolled back"
6. Tag a fix RC that removes the broken migrations
7. Apply the fix RC: verify it applies cleanly, binary self-updates

Test 2: Broken binary → self-verify rejection

1. Add os.Exit(1) to upgradeSelfVerifyCmd in cli/cmd/upgrade.go
2. Tag as a pre-release RC
3. Apply on a test server: the upgrade succeeds (migrations, health check) but self-verify fails
4. Verify: old binary is kept, service logs the failure, system continues running
5. Tag a fix RC that removes the os.Exit(1)
6. Apply the fix RC: verify binary self-updates successfully

What these tests exercise:

• Database rollback via rsync restore (Test 1)
• Binary self-update rejection on verify failure (Test 2)
• Version skipping: the fix RC supersedes the broken RC
• Service recovery: continues operating after failures

StatBus consists of five main services:

1. PostgreSQL: Database with Row Level Security and temporal tables
2. PostgREST: Automatic REST API from database schema
3. Caddy: Reverse proxy, auth gateway, and PostgreSQL TLS proxy
4. Next.js: Server-side rendered web application
5. Worker: Background job processor (Crystal)

See doc/service-architecture.md for detailed architecture.

1. User logs in via /rest/rpc/login (handled by PostgreSQL function)
2. JWT tokens stored in cookies (statbus and statbus-refresh)
3. Caddy extracts JWT from cookies and adds Authorization headers
4. PostgREST validates JWT and sets database role
5. Row Level Security enforces access control

Caddy provides secure direct PostgreSQL access:

Development Architecture:

psql → local.statbus.org:3024 (TLS+SNI)
  → Caddy (terminates TLS)
    → db:5432 (Docker network)

Benefits:

• TLS encryption without PostgreSQL TLS configuration
• SNI-based routing for multi-tenant deployments
• Standard tools (psql, pgAdmin, DBeaver) work seamlessly

See doc/service-architecture.md#postgresql-access-architecture for details.

Use tmp/ for development experiments:

• tmp/ - Backend scratch (SQL, scripts)
• app/tmp/ - Frontend scratch (TypeScript, configs)

These directories are gitignored but a pre-commit hook prevents accidental commits.

Backend Logs:

docker compose logs -f db      # PostgreSQL
docker compose logs -f rest    # PostgREST
docker compose logs -f proxy   # Caddy
docker compose logs -f worker  # Background worker

Frontend Debugging:

• Use Chrome DevTools
• React DevTools extension
• Next.js built-in error overlay

Database Debugging:

-- Enable query logging
ALTER DATABASE statbus_speed SET log_statement = 'all';

-- View recent queries
SELECT * FROM pg_stat_statements;

Before committing:

# Backend
./dev.sh test all

# Frontend
cd app
pnpm run lint
pnpm run format
pnpm run tsc
pnpm run test

See AGENTS.md for guidance on using AI coding assistants with StatBus.

• User Guide: For end users
• Deployment Guide: For administrators deploying single instance
• Cloud Guide: For SSB staff managing multi-tenant cloud
• Service Architecture: Technical details
• Integration Guide: API and PostgreSQL
• Conventions: Backend coding standards
• App Conventions: Frontend coding standards

• Issues: https://github.com/statisticsnorway/statbus/issues
• Discussions: https://github.com/statisticsnorway/statbus/discussions
• Website: https://www.statbus.org

1. Fork the repository
2. Create a feature branch (git checkout -b feat/my-feature)
3. Make your changes following conventions
4. Write/update tests
5. Commit with conventional commit messages
6. Push and create a Pull Request

Thank you for contributing to StatBus!