Skip to content
MalodeityPublic

About

No description, website, or topics provided.

Resources

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

forge

forge — stop solving the same data problems the hard way

Give Claude Code the vocabulary and judgment of a senior data platform engineer.

npx forge init

Install into any data repo in seconds. Works with dbt, Dagster, Prefect, Airflow, BigQuery, Snowflake, Redshift, Databricks, Kafka, Flink, and more.


The problem

Claude Code without domain context gives generic software engineering advice. It suggests "add error handling," "write tests," and "document your functions." None of that helps when the real problem is:

  • A new analyst defines "active users" differently than the existing dashboard — both are technically correct SQL, but they're measuring different things
  • A backfill overwrites the wrong partition scope and corrupts two weeks of fact data — silently
  • A dbt model ships with an ambiguous grain statement; nobody catches it until a board metric is wrong
  • A CDC pipeline quietly drops DELETE events and the staging table slowly diverges from source over three months
  • An ML feature is joined with post-prediction data — the model looks great in training and degrades in production
  • A schema migration renames a column with no downstream consumer inventory; four dashboards break in the same deploy
  • A pipeline incident fires at 2am and nobody can answer which partitions are corrupted or what depends on them
  • A GDPR deletion request arrives and nobody can trace which downstream tables contain that user's rows

These are not software engineering problems. They are data engineering problems — and Claude Code has no built-in knowledge of grain, CDC ordering, point-in-time correctness, deletion propagation, or data contracts.

forge fixes this.


How it works

forge installs a compact operating kit into your repo:

CLAUDE.md — domain knowledge A dense, production-grade standards block that encodes the principles that take years of data engineering experience to internalize: grain rules, SCD strategy, CDC patterns, data contracts, cost governance, GDPR/CCPA deletion propagation, and 10 hard rules with the "why" for each.

.claude/commands/ — 30 focused workflows Each command gives Claude a structured, domain-specific workflow for a category of data problem — from incident triage to cross-platform migration to compliance auditing. Claude stops giving generic advice and starts doing actual data engineering work.

.claude/references/ — warehouse, orchestrator, and artifact packs Commands can pull platform-specific guidance without bloating every prompt. Artifact templates turn a command run into a reusable output, not just commentary.

.forge/project.yaml — structured repo profile This is the normalized calibration file that /tune writes and the rest of forge reads. It keeps stack, command, layer, metric, and hard-rule context in one machine-friendly place.

Install-time trust profiles Choose standard, readonly, or strict-enterprise permission defaults during install so teams can start with the narrowest tool surface that still fits their workflow.

/tune — repo-specific calibration Run once after install. It scans your repo, detects your actual stack (dbt version, orchestrator, warehouse, ingestion tool, semantic layer), and writes repo-specific context into both CLAUDE.md and .forge/project.yaml. Future sessions know your stack, your model layer names, your entity vocabulary, and your exact dbt run commands — without you typing any of it.


Quick start

# 1. Install into your repo
npx forge init
npx forge init --settings-profile readonly

# 2. Open Claude Code
claude

# 3. Calibrate to this specific repo (run once)
/tune

# 4. Start working
/data          # broad health check — where should I start?
/incident      # something is actively broken
/model         # grain or modeling question
/metrics       # numbers don't agree
/lineage       # before any breaking change
/pii           # before shipping data to analysts

# 5. Validate the structured repo profile after manual edits
npx forge validate-profile --dir .

# 6. Generate a deterministic calibration baseline
npx forge sync-calibration --dir .

Commands

The command inventory below is generated from commands.manifest.json.

Find your footing

Start here when the repo is unfamiliar or a session needs fast calibration.

Command What it does
/context Orient a new session to the current data repo
/tune Detect the stack and write repo-specific calibration into CLAUDE.md and .forge/project.yaml

Diagnose and triage

Use these when the platform feels wrong, unstable, or hard to reason about.

Command What it does
/arch Review data architecture, seams, replayability, and coupling
/data Run a broad data platform health diagnostic and route to focused commands
/drift Detect schema, freshness, semantic, and contract drift before it becomes an incident
/incident Triage a broken pipeline from blast radius to recovery plan
/postmortem Turn a closed incident into root cause, corrective actions, and durable prevention
/monitor Design observability, alerts, SLAs, and on-call readiness for data systems

Model and define

Use these to lock down meaning before implementation drift turns into bad numbers.

Command What it does
/model Design or review facts, dimensions, marts, SCDs, and grain
/metrics Define KPI contracts and audit semantic correctness
/semantic Design or review semantic models, measures, dimensions, joins, and BI-facing metric layers
/contract Design producer-consumer data contracts with enforcement and compatibility rules
/catalog Audit metadata, ownership, glossary, lineage coverage, and catalog readiness

Move data safely

Use these when data movement or schema evolution can break downstream consumers.

Command What it does
/ingest Design or review CDC, batch, watermark, and stream ingestion pipelines
/lineage Trace lineage and blast radius before a breaking change
/schema Plan warehouse schema changes with compatibility and rollback safety
/reconcile Prove source-to-target correctness after backfills, migrations, or dual runs
/export Review reverse ETL, shared extracts, downstream contracts, and deletion propagation

Streaming and orchestration

Use these for event systems, scheduling, backfills, and physical execution shape.

Command What it does
/stream Review event contracts, ordering, lag, replay, and DLQ design
/orchestrate Review DAGs, assets, retries, scheduling, and backfill safety
/partition Design partitioning, clustering, sort keys, and incremental boundaries

Quality, cost, and compliance

Use these when correctness, privacy, cost, or test coverage are the limiting factor.

Command What it does
/quality Design a tiered data quality strategy with tooling and alerting
/optimize Optimize warehouse cost, query plans, materializations, and storage layout
/cost Assess warehouse spend, attribution gaps, and cost guardrails
/pii Audit PII handling, masking, deletion propagation, and residency
/features Review feature pipelines, leakage, point-in-time joins, and ML data safety
/dbt Review dbt project structure, testing, incremental correctness, and snapshots

Governance and platform evolution

Use these when the platform needs policy, access control, or large-scale change planning.

Command What it does
/govern Design governance, catalog policy, mesh ownership, retention, and compliance
/access Review RBAC, row and column security, and service account scope
/migrate Plan cross-platform warehouse, storage, or orchestration migrations

Command routing

You are facing… Start here
New to this repo /context → /tune
Broad health check, do not know where to start /data
Something broke in production /incident
Incident is closed and needs follow-through /postmortem
Suspicious change over time with no hard failure yet /drift
Weak observability or noisy alerts /monitor
Grain or modeling design question /model
Semantic layer or BI metric modeling review /semantic
Two dashboards show different numbers /metrics
Ownership and metadata audit /catalog
Planning a breaking schema change /lineage → /contract → /schema
Need to prove a migration or backfill is correct /reconcile
Designing a new ingestion pipeline /ingest
Reverse ETL or analyst extract review /export
Backfill or reprocessing needed /orchestrate
Streaming pipeline correctness /stream
Partitioning or clustering design /partition
Warehouse cost spike /cost → /optimize
Platform architecture review /arch
PII or privacy review /pii
Data quality from scratch /quality
dbt project review /dbt
Access control review /access
Governance or retention program design /govern
Warehouse or tool migration /migrate
ML model data preparation /features

For a shorter routing index, see docs/COMMAND_ROUTING.md.

Who it's for

Role What forge gives you
Data engineer Idempotent pipeline design, safe backfills, CDC ingestion, robust orchestration, incident recovery
Analytics engineer Grain correctness before shipping, semantic layer auditing, dbt contract enforcement
Analyst Canonical KPI definitions, divergence root cause, experiment metric correctness
Data scientist Leakage detection, point-in-time correctness, reproducible training sets, feature store design
Data platform engineer Architecture review, cost governance, lineage gaps, compliance posture

forge works in:

  • dbt projects on BigQuery, Snowflake, Redshift, Databricks, or DuckDB
  • Dagster, Prefect, Airflow, and dbt Cloud orchestration repos
  • Kafka, Kinesis, Pub/Sub, Flink, and Spark Structured Streaming repos
  • Lakehouse repos using Delta Lake, Apache Iceberg, or Hudi
  • ML feature pipeline repos and feature stores (Feast, Tecton, Hopsworks, Vertex AI)
  • Any repo where the primary work is data transformation, ingestion, or analytics

What gets installed

CLAUDE.md                  forge-managed standards block (prepended, idempotent)
.claude/settings.json      Pre-approved tool permissions
.claude/commands/          30 focused command files
.claude/references/        Warehouse, orchestrator, and artifact reference packs
.forge/project.yaml        Structured calibration file consumed by forge commands
.claudeignore              Excludes dbt_packages/, *.parquet, .dbt/, secrets

Running the installer twice is idempotent. The managed CLAUDE.md block is replaced in place; any custom content you've added below it is preserved. .forge/project.yaml is installed once and then left alone so /tune can own the repo-specific profile.

Pre-approved permissions include

dbt run/test/compile/docs · bq query/ls/show · aws s3 ls/cp/sync · gsutil ls/cp/cat · databricks runs/jobs/clusters · dagster asset/job/run · prefect deployment/flow · airflow dags/tasks · sqlmesh · standard git operations

Choose the trust profile at install time:

  • standard keeps the broad default engineering surface
  • readonly is for inspection-only sessions and blocks write-heavy actions
  • strict-enterprise keeps common validation commands while removing push and broader infra operations

Installation options

Option 1 — npm (recommended)

npx forge init

Pure Node.js. No shell scripts executed at runtime. All templates are bundled inside the npm package and verified by npm integrity checksums before installation. Works behind corporate npm proxies (Artifactory, Nexus). Does not trigger curl | bash security policies.

# Safer-by-default profiles for sensitive environments
npx forge init --settings-profile readonly
npx forge init --settings-profile strict-enterprise
# Update to the latest version at any time
npx forge update

# Validate the structured repo profile after manual edits
npx forge validate-profile --dir .

Option 2 — Manual (air-gapped or restricted environments)

git clone https://github.com/Malodeity/forge.git /tmp/forge

# Copy commands and settings into your repo
mkdir -p /your-repo/.claude /your-repo/.forge
cp -r /tmp/forge/.claude/commands /your-repo/.claude/
cp -r /tmp/forge/.claude/references /your-repo/.claude/
cp /tmp/forge/templates/settings.json /your-repo/.claude/settings.json
cp /tmp/forge/templates/project.yaml /your-repo/.forge/project.yaml

# Prepend CLAUDE.md content manually:
# cat templates/CLAUDE.base.md templates/CLAUDE.data.md
# Wrap in <!-- forge:begin --> / <!-- forge:end --> markers

No scripts executed. You review every file before it lands in your repo.

Option 3 — Shell installer (CI/CD pipelines)

curl -fsSL https://raw.githubusercontent.com/Malodeity/forge/main/install.sh | bash

# Pin to a specific version
curl -fsSL https://raw.githubusercontent.com/Malodeity/forge/v2.0.0/install.sh | bash

# Install into a specific directory
bash install.sh --dir /path/to/repo

⚠️ curl | bash executes arbitrary remote code. Use this only in CI/CD environments where your security policy permits it. For developer machines, use npm or manual install.

Options: --dir PATH · --version TAG · --stack data (legacy compatibility flag) · --settings-profile standard|readonly|strict-enterprise


Examples

Filled artifacts live under examples so teams can see what "good" looks like before they run commands in production repos.


Release and trust hardening

The repository now enforces a tighter maintenance and supply-chain posture:

  • manifest and reference inventory validation
  • packed-artifact verification before release
  • release tag checks against package.json and CHANGELOG.md
  • pinned GitHub Action SHAs
  • SBOM generation
  • npm provenance support in the publish workflow
  • security scans for CodeQL, dependency review, and secret scanning

Q&A

What does forge actually do once installed?

It gives your AI agent a data-engineering operating layer inside the repo: standards in CLAUDE.md, focused workflows in .claude/commands/, platform references in .claude/references/, and normalized repo context in .forge/project.yaml.

What kinds of problems is it meant to solve?

The recurring hard ones: broken backfills, CDC drift, schema migration risk, grain mistakes, metric disagreement, warehouse cost regressions, streaming correctness, deletion propagation, feature leakage, and architecture decisions that make systems hard to replay or govern.

How would a team use it in practice?

A common path is:

  • install forge
  • choose a trust profile if the environment is sensitive
  • run /tune once so the repo is calibrated
  • use /data for a broad diagnostic
  • branch into focused commands like /incident, /postmortem, /model, /metrics, /reconcile, /partition, or /export

That turns a generic agent into one that can reason in the repo's actual data vocabulary.

What makes it better than a single giant prompt?

A giant prompt gives static context. forge gives repeatable workflows with distinct scopes, artifact outputs, platform-specific references, and a shared project profile that other commands can reuse.

Is this only for dbt repos?

No. It is designed for broader data-platform work: dbt, orchestration repos, lakehouse repos, CDC and streaming repos, warehouse-heavy analytics repos, and ML-data pipelines.

Will this help with architecture, or just local fixes?

Architecture is a core goal. Commands like /arch, /migrate, /contract, /lineage, /partition, /monitor, and /govern are there to improve system shape, not just patch symptoms.


Security and data privacy

This is the section for teams asking the right hard question:

For the formal repository security policy and trust model, see SECURITY.md.

"If I install this into a repo with production logic, customer data, or regulated pipelines, is it going to leak anything?"

Short answer:

  • forge itself is not a hosted service
  • forge does not run a background daemon
  • forge does not include telemetry, analytics, or usage tracking
  • forge does not ship code that uploads your repo contents anywhere
  • the npm installer is local-only at runtime: it copies bundled files into your repo
  • the manual install path is fully inspectable before anything lands in your repo

What forge actually does:

  • installs markdown command files under .claude/commands/
  • installs reference files under .claude/references/
  • prepends a managed standards block into CLAUDE.md
  • creates .forge/project.yaml as a local calibration file for repo-specific context
  • installs .claude/settings.json and .claudeignore

What forge does not do by itself:

  • it does not scan your repo and phone home
  • it does not copy your code to a forge-owned API
  • it does not create a remote database of your models, schemas, or customer records
  • it does not retain your data on forge-controlled infrastructure, because there is no forge-controlled infrastructure in this package
  • it does not secretly export warehouse credentials, dbt profiles, or environment variables

Why this is safe by design

1. It is a file package, not a SaaS product

The core product is a set of local files plus a local installer. When you use npx forge init, the Node CLI copies files already bundled in the package into your repo. It is not a thin client talking to a forge backend.

2. The npm path does not fetch remote templates at runtime

The recommended install path is the Node CLI. Its templates are bundled into the npm package and copied locally with fs. That means the install path does not need to fetch your repo contents or send them to an external forge service.

3. The shell installer is explicit about its tradeoff

install.sh can download the package files from GitHub when used as curl | bash or with --version, which is why it is documented as a CI/CD-oriented option with an explicit warning. That is a supply-chain trust decision, not a data exfiltration channel built into forge logic.

4. No telemetry or analytics layer exists in this repo

There is no usage tracking client, event emitter, analytics SDK, or background collector in the package. The repo does not contain code for sending command usage, repo metadata, or data samples to a forge endpoint.

5. Retention is local and obvious

forge does not maintain a server-side memory or cloud retention layer. The only retained state it creates is local repo state such as:

  • CLAUDE.md
  • .claude/commands/
  • .claude/references/
  • .forge/project.yaml

If you remove those files, forge has nothing left to retain.

The real trust boundary

It is important to be precise here: forge can make an AI agent more capable inside your repo, but the AI runtime you choose still matters.

If you use forge with an assistant that sends prompts or file context to a remote model provider, then your privacy posture depends on:

  • that provider's data handling policy
  • the tool permissions granted to the agent
  • what files you let the agent read
  • what commands you let the agent execute

So the accurate statement is:

  • forge itself is not stealing data
  • forge itself is not a telemetry pipeline
  • forge itself does not introduce a hidden retention system
  • but your AI assistant, model provider, and tool configuration still define the broader privacy boundary

What lives in .forge/project.yaml

.forge/project.yaml is meant to hold normalized repo context such as:

  • stack names like dbt, airflow, snowflake
  • common repo commands
  • layer names like staging, marts, exports
  • entity and metric names
  • repo-specific hard rules

It should not be used to store:

  • secrets
  • API keys
  • passwords
  • tokens
  • raw customer records
  • regulated data extracts

Treat it like checked-in project metadata, not a secret store.

What about customer data in prompts?

That risk is not unique to forge; it exists any time an agent reads files or query results. forge helps by pushing users toward better boundaries:

  • /pii audits masking, deletion propagation, and residency
  • /access reviews who can see what
  • /export challenges whether sensitive data should leave the warehouse
  • /contract and /govern push ownership and policy clarity

In other words, forge is intended to reduce sloppy data handling, not normalize it.

Recommended safe usage for sensitive environments

  • Prefer npx forge init or manual install over curl | bash on developer machines
  • Prefer --settings-profile readonly or --settings-profile strict-enterprise in regulated or trust-sensitive environments
  • Review .claude/settings.json before using the package in a regulated environment
  • Do not place secrets in CLAUDE.md or .forge/project.yaml
  • Run /tune against repo structure and commands, not copied production datasets
  • Use masked samples or metadata-only inspection when working around PII-heavy systems
  • Pair forge with your organization's approved AI provider and retention policy

Security Q&A

Does forge upload my code to its own servers?

No. There is no forge-operated upload service in this repo. The npm installer copies local package files into your repo. The manual path is fully inspectable. The shell installer downloads package files from GitHub, but that is still code delivery, not repo-content upload.

Does forge store my schema, tables, or customer records anywhere?

Not on forge infrastructure. forge only creates local repo files. It has no backend retention store. Any separate retention behavior would come from the AI tool or provider you choose to use with it.

Is forge reading my warehouse behind the scenes?

No. forge does not run background jobs or hidden queries. An agent may run commands later if you use the installed workflows with permissions that allow it, but that is explicit agent behavior in your environment, not hidden forge telemetry.

Can forge leak secrets from my repo?

Not by itself. It does not contain logic to export secrets. But any AI agent with shell or file access can still be dangerous if you grant too-broad permissions. The safe posture is to treat assistant permissions, provider settings, and secret hygiene as first-class controls.

Does forge keep data after I uninstall it?

Only whatever local files remain in your repo. Delete the installed files and there is no forge-managed server-side copy to clean up.

Should I use forge in healthcare, finance, or other regulated environments?

Yes, with the same review standard you would apply to any repo tool that influences an AI agent. The package design is local and inspectable, but regulated use still depends on your approved model provider, internal access controls, retention policy, and command permissions.


Standards reference

The installed CLAUDE.md block encodes production-grade standards across 14 domains:

Domain What's covered
Architecture patterns Medallion, star schema, Data Vault, event-first, feature store — when to use each
Modeling rules Grain, SCD types, bridge tables, surrogate vs business keys, null semantics
Metrics and semantics KPI contracts, ratio correctness, distinct count safety, snapshot joins
Data contracts Compatibility matrix (BACKWARD/FORWARD/FULL), producer/consumer responsibilities
Ingestion and CDC Pattern selection, offset management, DELETE capture, schema drift handling
Orchestration and recovery Partition reruns, backfill planning, late data, asset vs task DAGs
Streaming Delivery semantics, watermarks, DLQ, replay, partition key discipline
Query, storage, cost EXPLAIN-first, partitioning, materialization, small-file risk, storage formats
Cost governance Cost attribution tagging, query byte limits, autoscaling ceilings
Data quality Freshness SLAs, null rate checks, reconciliation, quality gate taxonomy
Observability and SLAs What every pipeline must publish: status, rows, partition, duration
Security and compliance PII classification, right to erasure (GDPR/CCPA), data residency
Testing strategy Unit, model, pipeline, contract, streaming, integration — by layer
Hard rules 10 non-negotiable rules with the "why" for each

Development

# Run the full test suite
npm test

# Regenerate manifest-driven docs
npm run generate:docs

# Verify generated docs are in sync
npm run check:docs

# Verify the published npm tarball installs correctly
npm run verify:pack

# Validate the current repo profile shape
npm run validate:profile

# Syntax-check the shell installer
bash -n install.sh

# Validate JSON templates
python3 -m json.tool templates/settings.json
python3 -m json.tool templates/settings.readonly.json
python3 -m json.tool templates/settings.strict-enterprise.json
python3 -m json.tool .claude/settings.json

# Smoke test the npm CLI
node bin/forge.js init --dir /tmp/forge-cli-smoke
node bin/forge.js init --dir /tmp/forge-cli-smoke-ro --settings-profile readonly
node bin/forge.js validate-profile --dir /tmp/forge-cli-smoke

# Smoke test the shell installer
bash install.sh --dir /tmp/forge-sh-smoke
bash install.sh --dir /tmp/forge-sh-smoke-ro --settings-profile readonly
bash install.sh --dir /tmp/forge-sh-smoke --stack data

# Smoke test install into a fixture repo
cp -R fixtures/dbt-bigquery /tmp/forge-fixture-bigquery
bash install.sh --dir /tmp/forge-fixture-bigquery

# Verify idempotency (second run must produce identical output)
bash install.sh --dir /tmp/forge-idem
cp /tmp/forge-idem/CLAUDE.md /tmp/claude-first.md
bash install.sh --dir /tmp/forge-idem
diff /tmp/claude-first.md /tmp/forge-idem/CLAUDE.md && echo "✓ idempotent"

CI validates on every push: installer regression tests, manifest-driven command inventory checks, JSON validation, shellcheck, fixture-backed install smoke tests, template size limits, and idempotency.

Adding or renaming a command

If you add, rename, or remove a command, update these in the same PR:

  1. commands.manifest.json — source of truth for the command inventory
  2. .claude/commands/ — the command file itself
  3. README.md — public command docs
  4. tests/installer.test.js — installer and standard coverage
  5. .github/workflows/validate.yml — CI expectations

For command selection help, see docs/COMMAND_ROUTING.md. For the required shape of each command file, see docs/COMMAND_STANDARD.md.


License

MIT — use freely in any project, commercial or otherwise.

About

No description, website, or topics provided.

Resources

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages