This directory holds the automated tests for HydroModPy. It is split into five release tiers plus focused auxiliary tiers with different purposes, budgets, and selection rules.

tests/
├── conftest.py                  # shared scratch root, update-goldens flag, shared fixtures
├── unit/                        # one module under test, 2 s target, no real I/O
├── integration/                 # cross-module workflows with shared fixtures, < 10 s
├── regression/                  # golden-reference tests for full workflows
│   ├── fast/                    # routine non-regression tier
│   ├── extensive/               # deeper end-to-end non-regression tier
│   └── reference/               # committed golden JSON files
├── validation/                  # scientific benchmarks (analytical, MMS)
│   ├── analytical/
│   │   ├── steady/
│   │   └── transient/
│   └── numerical/
├── characterization/            # what a produced run offers a reader, plus the
│                                # double-execution comparator
├── e2e/                         # subprocess-level command scenarios
├── performance/                 # pytest-benchmark storage-wrapper benchmarks
├── contract/                    # compatibility contracts across implementations
└── _helpers/                    # pytest-local helpers and shared builders

validity_frame/ is a separate installable package with its own validity_frame/tests/; it is not part of this suite.

Declared in the repository-root pytest.ini:

The fast/extensive markers under tests/regression/ are auto-applied based on the file's subdirectory (see tests/conftest.py).

# Unit tests (fastest feedback)
pytest tests/unit/ -q                         # all unit tests
pytest tests/unit/solver -q                   # a subpath
pytest tests/unit/ -q -k metrics_nse          # keyword filter

# Regression - fast tier (parallel)
pytest tests/regression/fast/ -q -n auto
pytest -m "regression and intercomparison" -q -n 1

# Regression - extensive tier (serial)
pytest tests/regression/extensive/ -q -n 1

# Validation - all analytical cases
pytest tests/validation/ -q

# Characterization - one real run, read back from disk
pytest tests/characterization/ -q

# Double-execution comparator on the four frozen projects (about one minute)
python -m tests.characterization.comparator --frozen --report ~/hmp-parity.md

# Marker selection
pytest -m "regression and fast" -q
pytest -m "validation and steady" -q
pytest -m nwt -q
pytest -m "not slow" -q

# Parallel execution (xdist)
pytest tests/unit/ -q -n auto
pytest tests/regression/fast/ -q -n auto

The hmp CLI wraps the most common invocations:

hmp test unit                # → pytest tests/unit/
hmp test regression --fast   # → pytest tests/regression/fast/
hmp test regression --fast --intercomparison
hmp test regression --extensive
hmp test regression --update-goldens

Scratch data lands in a repository-external directory to keep the working tree clean:

The conftest sets TMPDIR/TMP/TEMP to a subdirectory of the scratch root so spawned subprocesses inherit the same cleanup policy.

Regression tests compare the output of a pipeline run to a committed signature - a statistical summary of fields stored as JSON under tests/regression/reference/golden_references/{fast,extensive}/.

When a regression assertion fails:

1. Identify the golden file referenced by the failing test (path is printed in the assertion message).
2. Open the corresponding JSON and compare the stored signature to the freshly computed one.
3. Decide whether the difference is:
   • a legitimate code change - update the golden (see below),
   • a numerical drift - investigate the responsible commit,
   • a platform difference - document and, if needed, tighten the tolerance or mark the test platform-specific.

Never edit a JSON file by hand. Use the dedicated CLI switch, which regenerates the golden from the current output:

# Refresh every golden referenced by a test run
pytest tests/regression/ -q -n auto --update-goldens

# Refresh the goldens of a single test module
pytest tests/regression/fast/test_simulation_regression_fast_mf6_regression.py \
    -q --update-goldens

Then commit the updated JSON files in a dedicated commit. The commit message must state the scientific or code-level reason the signature changed.

Numerical comparisons use numpy.testing.assert_allclose (or equivalent math.isclose) with an explicit rtol/atol. Prefer statistical signatures (min, p25, p50, p75, p95, max, mean, std, sum) over full-array dumps so goldens stay compact and cross-platform stable.

Validation tolerances live in tests/TOLERANCES.md, with some case-local files under validation_cases/**/tolerances.toml when the benchmark owns a named envelope. tests/_helpers/tolerances.py::tol() parses the shared table for tests that need a named tolerance.

• unit/ - one importable module under test, one behaviour per test, 2 s target. The hard limit enforced by tests/conftest.py is a 60 s per-test timeout; anything that needs more must carry @pytest.mark.slow, which takes it out of the push gate. No external binaries. Use tmp_path for any I/O.
• integration/ - cross-module test (pipeline + catalog, planner + adapters, …). Budget ≤ 10 s, no golden files. Two shared fixtures are available from the root conftest, tmp_workspace and minimal_config; neither is mandatory.
• regression/ - exercise a full launcher / pipeline on a fixture, then compare a committed signature. Tag with @pytest.mark.regression and (if solver-specific) @pytest.mark.nwt or @pytest.mark.mf6.
• validation/ - exercise a physical case against a known analytical solution; document the reference and the tolerance rationale.

The shared fixtures come from tests/conftest.py:

• tmp_workspace(tmp_path) yields an initialized workspace directory (standard data/, projects/, one data/<variable>/ folder per variable) ready to back a SimulationCatalog.
• minimal_config(tmp_path) returns the smallest valid HydroModPyConfig (synthetic geographic + a workspace pointed at tmp_path / "project"). Extend via .model_copy(update=...).

Prefer @pytest.mark.parametrize over copy-pasting N near-identical tests. Keep test files under ~400 LOC: a file that grows beyond that is usually hiding several independent suites.

The active GitHub Actions gates are split across .github/workflows/:

• main-ci.yml - PR + push gate: quality (ruff), secrets, architecture, a --collect-only gate over the whole tree, fast / unit (-m "not slow") / contract / integration / regression-fast tiers, package-smoke, typing.
• ci-nightly.yml - nightly extensive regression, validation (not petsc), integration with coverage.
• ci-weekly.yml - cross-OS unit + fast-regression + analytical, serial e2e, full regression.
• perf.yml - pytest-benchmark drift gate over tests/performance/.
• petsc-smoke.yml - Linux PETSc Boussinesq smoke subset (conda env).

Validation, extensive regression, and e2e are not part of the fast PR-blocking suite; run validation manually before a release with pytest tests/validation/ -q.