kardashev-data

Self-hosted US energy grid data platform. Ingests real-time and historical datasets from ISO/EIA/ERCOT/CAISO and other public sources into Postgres, serves them through a read-only REST API, and powers the Kardashev Labs tools.

Live API: https://data.kardashevlabs.org (interactive docs)

No authentication required.

# Real-time carbon intensity (lbs CO2/MWh) for all US ISOs
curl https://data.kardashevlabs.org/carbon/latest

# Live LMP prices at NYISO zone hubs
curl "https://data.kardashevlabs.org/lmp?iso=NYISO&market=RT&limit=20"

# Nodal price map: every priced node in ERCOT right now
curl "https://data.kardashevlabs.org/lmp/map?iso=ERCOT&market=RT"

# Fuel mix across all 7 ISOs (last 24h)
curl "https://data.kardashevlabs.org/fuel-mix?iso=CAISO"

# Solar + wind curtailment in California (last 30 days)
curl "https://data.kardashevlabs.org/curtailment?iso=CAISO&days=30"

# Henry Hub nat gas price history
curl "https://data.kardashevlabs.org/natural-gas/latest"

LMP data also has a standalone historical backfill path (ingest/backfill_lmp.py, not part of the live scheduler) that pulls ERCOT MIS yearly DAM/RTM hub + load-zone price files and CAISO OASIS PRC_LMP/PRC_INTVL_LMP to build out multi-year history for the forecasting dataset. LMP retention in the live table defaults to 10 years (LMP_RETENTION_DAYS).

ISO APIs / EIA / ERCOT CDR / CAISO OASIS / NYISO Open Data
        ↓ (Python ingest workers)
   PostgreSQL (time-series schema)
        ↓ (FastAPI, asyncpg)
   REST API  →  kardashevlabs.org tools

Stack:

• Python 3.12, FastAPI, asyncpg
• PostgreSQL (Railway managed)
• Ingest scheduler: ingest/scheduler.py, a simple while True loop with per-job cadence tracking (no external scheduler library)
• kardashev package (separate PyPI package, pinned in pyproject.toml) for direct ISO data access (CAISO, ERCOT, MISO, NYISO, ISONE, SPP, PJM); iso_data/ in this repo retains only supplementary, non-ISO sources (BPA, NOAA weather, NRC, EPA CAMPD, RGGI, USBR, NREL, EIA-930/commodities)
• ingest/backfill_lmp.py: standalone CLI for historical LMP backfill (ERCOT MIS yearly files, CAISO OASIS), run manually/out-of-band rather than on the live scheduler
• Railway (API service + ingest worker, two services sharing one Dockerfile, both auto-deploy from main)

Data sources: CAISO OASIS, ERCOT CDR/MIS, NYISO Open Data, MISO API, SPP Marketplace, PJM Data Miner, ISO-NE web services, EIA Open Data API, NRC daily status, EPA CAMPD API, USBR HydroMet, USGS Water Services, NREL NSRDB, NOAA weather, RGGI/CA ARB auction results.

GIS ingestion preserves each source document and extraction in ercot_gis_filings and its project rows in ercot_gis_observations. Filings include ERCOT's document ID, download URL, filename, publication timestamp, SHA-256, exact downloaded bytes (raw_file), parser version, and extraction timestamp. filing_id joins the source file to every extracted row; project identity remains ERCOT's INR (queue_id).

The scheduler checks daily at 16:00 UTC. Each available file is downloaded and hashed, including previously seen document IDs, so changed bytes under the same ID are detectable. Identical document ID/publication/hash/parser combinations are skipped. This intentionally uses more bandwidth than checking only filenames. Increment PARSER_VERSION in ingest/ercot_gis.py when correcting extraction logic; the next run stores a new extraction while retaining the earlier one. A failed or empty parse leaves existing history intact and is retried on a later run.

ercot_gis_snapshots remains the compatible monthly projection for existing readers. It selects the latest publication for each month; extraction time and filing ID break publication-time ties. Replacement is atomic and removes projects omitted by a corrected filing. Older backfills cannot supersede later publications. Pre-migration rows are retained separately in ercot_gis_legacy_snapshots: they have no verified source-document provenance and are not represented as new auditable filings. A migration alone cannot recover previously missed revisions.

Rollout (in the deployed environment with its configured database):

python -m db.migrate
python -m ingest.ercot_gis --backfill
python -m ingest.ercot_gis_timelines

The backfill checks all files still available from ERCOT; it cannot recover files ERCOT has removed or overwritten before they were archived. Both normal and --backfill runs now inspect all available documents. --dry-run --limit 1 checks extraction without accessing the database.

Inspect a project's source-backed history:

SELECT f.filing_id, f.source_document_id, f.source_url, f.published_at,
       f.extracted_at, f.content_sha256, f.parser_version,
       o.snapshot_month, o.queue_id, o.capacity_mw, o.projected_cod
FROM ercot_gis_observations o
JOIN ercot_gis_filings f USING (filing_id)
WHERE o.queue_id = 'YOUR_ERCOT_INR'
ORDER BY f.published_at, f.extracted_at, f.filing_id;

Retrieve the exact source bytes with SELECT raw_file FROM ercot_gis_filings WHERE filing_id = .... Publication time describes ERCOT's metadata; extraction time describes when this system captured that version. For a historical "what had Kardashev captured by then" query, constrain both timestamps. The monthly projection and existing aggregate metrics are not an as-of audit API.

Storage regression tests use isolated schemas in a disposable PostgreSQL database:

GIS_TEST_DATABASE_URL=postgresql://localhost/kardashev_test pytest tests/test_gis_filing_storage.py -q

python -m venv .venv && source .venv/bin/activate
pip install -e ".[api,ingest,dev]"
cp .env.example .env          # fill in DATABASE_URL and EIA_API_KEY

Docker (API + ingest + Postgres all-in-one):

docker compose up --build

Or against a local Postgres:

python -m db.migrate           # create schema
uvicorn api.main:app --reload  # API at localhost:8000
python -m ingest.scheduler     # start data collection

Interactive docs at http://localhost:8000/docs.

Historical LMP backfill (separate from the live scheduler):

python -m ingest.backfill_lmp ercot --start-year 2019
python -m ingest.backfill_lmp caiso --start 2024-01-01 --end 2026-07-06

.env.example in this repo covers the API-service variables above; ISO keys and anomaly webhook/email vars belong on the ingest service.

pytest -q

MIT