SNANA Pipeline Assistant

An LLM-powered operations assistant for SNANA/Pippin pipelines. Automatically diagnoses pipeline failures (stale locks, cached config mismatches, out-of-memory errors) using a curated, structured knowledge base and NERSC/Slurm scheduler state. Follows operational-first debugging discipline to rule out simple causes before speculating about code-level bugs.

pip install --upgrade pip          # older cluster pip (<21.3) chokes on the build
pip install "isnana[all]"          # or isnana[anthropic] / [openai] / [gemini] for just one SDK

snana-assistant init

# Set your API key (supports ANTHROPIC_API_KEY, OPENAI_API_KEY, or GEMINI_API_KEY)
export ANTHROPIC_API_KEY="your-api-key"

snana-assistant diagnose "BBC aborts citing sigint, tried sigint_fix, still fails"

The package installs as isnana; it provides both an isnana and a snana-assistant command (identical). Knowledge-base and SNANA-manual search work offline with no key; diagnose needs an API key or a local Ollama model.

To run on Midway or other clusters, clone and install locally.

First, ensure you are using Python 3.10 or higher. You can set up your environment using either Conda or a standard virtualenv:

# Create and activate a Python 3.10 environment
conda create -n snana_env python=3.10 -y
conda activate snana_env

# Clone and install the package
git clone https://github.com/am610/SNANA_PIPELINE_ASSISTANT.git
cd SNANA_PIPELINE_ASSISTANT
pip install --upgrade pip
pip install -e .[all]

git clone https://github.com/am610/SNANA_PIPELINE_ASSISTANT.git
cd SNANA_PIPELINE_ASSISTANT

# (Optional: load python/3.10 if default is older)
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip  # Crucial on older cluster environments
pip install -e .[all]

snana-assistant init

# Set your API key (supports ANTHROPIC_API_KEY, OPENAI_API_KEY, or GEMINI_API_KEY)
export ANTHROPIC_API_KEY="your-api-key"

snana-assistant diagnose "BBC aborts citing sigint, tried sigint_fix, still fails"

In future cluster logins, you can run the assistant from any directory (it does not need to be the cloned repository directory) by running:

# If you set up using Conda:
conda activate snana_env

# If you set up using a virtualenv:
source /path/to/SNANA_PIPELINE_ASSISTANT/.venv/bin/activate

# (Optional) Auto-load your API key every session by appending it once to ~/.bashrc:
# echo 'export ANTHROPIC_API_KEY="your-api-key"' >> ~/.bashrc
# snana-assistant diagnose "BBC aborts citing sigint, tried sigint_fix, still fails"

The SNANA Pipeline Assistant provides one unified, canonical diagnostic capability accessible across multiple environments:

                         SNANA Pipeline Assistant
                                   |
                  Canonical Domain Procedure & Safety
                     (agent-skill/snana-assistant/)
                                   |
                 +-----------------+-----------------+
                 |                                   |
                 v                                   v
      AI Coding Agent Skills              Deterministic Read-Only MCP
   (Claude Code / Codex / Gemini)           (snana-assistant-mcp)
                 |                                   |
                 +-----------------+-----------------+
                                   |
                                   v
                        Standalone CLI & Eval
                         (isnana / snana-assistant)

Install the snana-assistant Agent Skill and read-only MCP server into whichever AI assistant you already use. The agent follows the same operational-first debugging sequence and safety guarantees across models.

Add the repository marketplace and install the plugin:

/plugin marketplace add am610/SNANA_PIPELINE_ASSISTANT
/plugin install snana-assistant@snana-tools

Or install the standalone skill directly: copy agent-skill/snana-assistant to ~/.claude/skills/snana-assistant.

Install the canonical skill using the included helper:

# Global install (~/.agents/skills/snana-assistant):
./integrations/codex/install-skill.sh --user

# Or repository-scoped (.agents/skills/snana-assistant):
./integrations/codex/install-skill.sh --repo

Then register snana-assistant-mcp in your Codex MCP configuration. See integrations/codex/README.md.

Install the skill directly from Git or local checkout:

gemini skills install https://github.com/am610/SNANA_PIPELINE_ASSISTANT.git --path agent-skill/snana-assistant
# Or link locally:
gemini skills link ./agent-skill/snana-assistant

Then register the read-only MCP server:

gemini mcp add snana-assistant --command snana-assistant-mcp

See integrations/gemini/README.md.

Run non-interactively in scripts, cron jobs, CI pipelines, or local offline environments using the pinned CLI (isnana or snana-assistant). Backed by automated regression evaluation suites (cases.yaml).

All AI agent integrations and MCP tool endpoints (snana-assistant-mcp) enforce strict read-only safety policies:

• No file modifications or deletions: Never alters configs or deletes lock files automatically.
• No job submissions or cancellations: Strictly read-only scheduler checks (squeue -u $USER / qstat).
• No arbitrary shell execution: No unconstrained shell access.
• Bounded queries & secret suppression: Capped log and file reads; sensitive tokens, keys, and credentials are never exposed.

You don't need to know a path, or paste an ls. The assistant can list directories and grep file contents itself, so questions like these work directly:

snana-assistant diagnose "check the files in this directory and see which script uses sim_ia_salt_des5yr.input"
snana-assistant diagnose "where is DES-SN5YR_DES.HOSTLIB referenced?"

It will list_directory to see what exists, search_files to find which script or YAML references the file, then read the hits to confirm — rather than inferring from naming convention or asking you to supply the path.

Searches are bounded so a Pippin output tree can't swamp the answer: 200 entries per listing, 50 matches per search, 4 levels deep, binaries/FITS/gzip skipped, symlinks not followed. Anything truncated is reported.

diagnose answers one question and forgets it. For anything that takes a few follow-ups, use chat — the conversation is carried across turns, so the assistant remembers what you asked, what it answered, and which files it already read:

snana-assistant chat

you> check sim_ia_salt_des5yr.input -- what does it do and what does it depend on?
...
you> what about that WGTMAP path, will it resolve under Pippin?
you> /reset     # start a fresh conversation
you> /exit      # quit (Ctrl-D also works)

Long sessions accumulate context — every file and manual chunk the assistant read stays in the conversation and is resent each turn, which costs tokens. /reset when you move to an unrelated problem.

Programmatic equivalent:

from snana_assistant.agent import Agent

session = Agent().session()
session.ask("what does sim_ia_salt_des5yr.input do?")
session.ask("and what was the GENVERSION in it?")   # remembers the file

Besides diagnosing crashes, diagnose will read and review a config/input file you point it at — what it does, what it depends on, and whether any keys look wrong:

snana-assistant diagnose "check sim_ia_salt_des5yr.input -- is it written correctly, \
  what dependency files does it call, and what is it supposed to do?"

These reviews cost more agent turns than a failure lookup, because the assistant searches the knowledge base, gotchas, and manual before it opens the file, then follows any INPUT_FILE_INCLUDE chain. If an answer ever comes back tagged

[incomplete: stopped after 15 tool-use turns without reaching a final answer -- rerun with a higher --max-turns]

rerun with a bigger budget:

snana-assistant diagnose "..." --max-turns 25 --max-tokens 8192

graph TD
    User([User Query]) --> Agent[Agent Loop]
    Agent --> KB[BM25 + Morphological Knowledge Base]
    Agent --> Tools[CLI Diagnostic Tools]
    Tools --> Slurm[check_job_status]
    Tools --> Config[diff_config]
    Tools --> Logs[read_log_tail]
    Tools --> Manual[search_manual]
    KB --> Result[\[entry-id\] Cited Cause + Fix]

1. Scheduler State: Checks Slurm queue for completing (CG), pending (PD), or job-name truncation conflicts.
2. Config Staging: Compares your source configuration against Pippin's output staging directory copy to catch cached config mismatches.
3. Log Scanning: Tails execution logs and parses standard patterns (OOM, killed, timeout, segmentation faults).
4. Knowledge Search: Queries a compiled, structured database of historical failure modes.
5. LaTeX Manual Index: Fallback search over section-chunked SNANA manual LaTeX files.

docker pull ghcr.io/am610/snana-pipeline-assistant:latest
docker run --rm --env-file .env ghcr.io/am610/snana-pipeline-assistant \
  diagnose "BBC aborts citing sigint, tried sigint_fix, still fails"

On HPC systems without a Docker daemon, Singularity/Apptainer can run the same image directly: singularity run docker://ghcr.io/am610/snana-pipeline-assistant:latest diagnose "...". Built and pushed automatically on every change to main (see .github/workflows/publish-image.yml) — always current with this repo.

Answers stream as they are written, so you see output in ~2s rather than waiting for the whole reply. Pipe or redirect the output and it reverts to a single clean block for scripts; --no-stream forces that explicitly.

The prompt prefix (system prompt + tool schemas + conversation so far) is cached between turns, which cuts input-token cost roughly in half — measured 10,825 → 5,570 billed input equivalents on a 3-turn query. Savings are largest in chat, where history compounds. Cache entries expire after ~5 minutes, so an occasional one-shot query pays a small write surcharge without benefiting; set SNANA_ASSISTANT_NO_CACHE=1 to disable.

Most remaining latency is simply the number of sequential model calls an investigation needs — each tool round trip is 1.5–3.5s. Raise or lower it with --max-turns.

DOE/HPC users wary of sending logs to external APIs can configure the tool to run entirely offline with a local model:

1. Start Ollama locally: ollama run llama3
2. Run snana-assistant init and select the local model as your default backend.
3. Knowledge Base searches and SNANA LaTeX manual lookups run fully offline, zero-key, immediately after install.

If the assistant cannot diagnose your issue, it logs the uncaptured query locally. You can generate a pre-filled, templated GitHub issue to submit the new failure mode for verification:

snana-assistant feedback

This command URL-encodes your query and generates a draft issue link, keeping your actual data private unless you choose to submit it.