HICode Alternative

This repo is an alternative implementation of the method presented in the HICode: Hierarchical Inductive Coding with LLMs paper by Mian Zhong, Pristina Wang & Anjalie Field. The intent is making it more practical to run, and providing a rich visualization dashboard.

This implementation introduces some differences that may yield different results. For the original implementation, refer to the Original Repo. This is similar enough for personal use, but not for repoduction or evaluation.

Run hierarchical inductive coding on any CSV text corpus, then inspect the resulting themes and source evidence. The caller supplies the CSV, the text column, and a research-specific prompt profile.

Run commands from the repository root. Input, output, prompt, and run identity parameters are required. Processing limits and concurrency settings have built-in defaults.

The repository is organized as follows:

data/                    example source datasets
configs/prompt_profiles/ prompt profiles used by the pipeline (ignored by gitignore)
src/hicode/              importable pipeline and dashboard code
tests/                   automated tests
docs/                    paper and methodology notes
output/                  local generated run artifacts (ignored by gitignore)

There is an example run for clothing reviews included in the repo.

uv sync

Create .env:

cp .env.template

Then, edit with your own credentials and settings.

The checked-in example CSV has a single Review_Text column. Any CSV can be used by passing the name of its text column with --text-column.

uv run hicode \
  --input-csv data/clothing_reviews.csv \
  --text-column Review_Text \
  --run-name smoke-30 \
  --mode new \
  --runs-dir output/runs \
  --prompt-file configs/prompt_profiles/clothing_reviews.json \
  --max-usable-records 30 \
  --cluster-stop stable

Prompt profiles are deliberately explicit because the research question and coding instructions vary by corpus. The included profiles are examples and can be replaced with any JSON profile matching the documented schema. configs/prompt_profiles/template.json is a paper prompt template, not a ready-to-run profile. Its background and research-question fields intentionally contain <ADD YOUR OWN: ...> placeholders. Replace both with the corpus-specific components specified by the paper before running; HiCode rejects the unchanged template.

You can also alter the rest of the prompt profile to deviate further from HICode if desired.

Use a new run name. By default, the full corpus is processed:

uv run hicode \
  --input-csv path/to/your_records.csv \
  --text-column body \
  --run-name analysis-v1 \
  --mode new \
  --runs-dir output/runs \
  --prompt-file configs/prompt_profiles/your_profile.json \
  --cluster-stop stable

--max-usable-records can impose a limit after blank values are skipped.

Choose exactly one clustering stop mode. These concise forms show the required limits for each mode:

# Stop when an iteration produces the same theme count as its input.
--cluster-stop stable

# Stop after the specified maximum number of iterations.
--cluster-stop max-iterations --max-cluster-iterations N

# Stop when the result reaches the target; this mode has no iteration cap.
--cluster-stop target-themes --target-final-themes N

# Stop at whichever configured limit is reached first.
--cluster-stop max-iterations-or-target-themes \
  --max-cluster-iterations N --target-final-themes N

Pass the same input, text column, prompt profile, run name, and algorithm parameters again with --mode resume:

uv run hicode \
  --input-csv path/to/records.csv \
  --text-column body \
  --run-name analysis-v1 \
  --mode resume \
  --runs-dir output/runs \
  --prompt-file configs/prompt_profiles/your_profile.json

A fresh run never overwrites an existing run, and a completed run cannot be resumed. Worker counts may change when resuming; completed checkpoints are reused. Algorithm parameters are part of the run configuration and must remain the same when resuming.

HiCode presents each run as a compact five-stage terminal narrative. The run header shows the run name, mode, input and output paths, profile, models, reasoning effort, worker counts, corpus limit, and clustering stop rule. The numbered stages are shown once:

1. Preprocessing reports included and excluded records and segment counts.
2. Generation shows live label progress with completed/total units, percent, elapsed time, throughput, and an ETA for the remaining segments.
3. Clustering shows one live batch-progress bar per hierarchy level, followed by input-label, theme, reduction, elapsed-time, and stop-condition details.
4. Assignments reports the assignment totals.
5. Export reports the output destination.

Context and active work use cyan or blue, completed work uses green, resume and retry warnings use yellow, and failures use red. These colors communicate status only; HiCode does not add decorative terminal output. Resumed preprocessing, generation, and clustering work is marked as resumed and seeds progress from its checkpoints rather than being replayed as new work.

The final success summary includes total elapsed time, records, segments, relevance, code occurrences, unique codes, hierarchy levels, final themes, the stop reason, and the output directory. Failed or interrupted runs show a concise status after their execution metadata is updated.

ETA is scoped to work whose total is known: remaining generation segments or the current clustering level's remaining batches. For stable and target-themes stop modes, the number of hierarchy levels is unknown, so the terminal does not claim a whole-run ETA; an open-ended level label is shown instead. Bounded modes identify the configured level limit in the progress label.

When output is redirected, Rich automatically emits readable plain text without ANSI color codes or continuously redrawn terminal frames. Set the standard NO_COLOR environment variable to opt out of color explicitly:

NO_COLOR=1 uv run hicode --input-csv data/records.csv --text-column body \
  --run-name analysis-v1 --mode new --runs-dir output/runs \
  --prompt-file configs/prompt_profiles/your_profile.json \
  --cluster-stop max-iterations --max-cluster-iterations 5

uv run streamlit run dashboard.py -- \
  --input-dir output/runs/analysis-v1

Already recorded example (clothing reviews):

uv run streamlit run dashboard.py -- \
  --input-dir output/runs/clothing_reviews

The dashboard is read-only and makes no API calls.

uv run hicode [options]

HiCode normalizes Unicode, line endings, control characters, and whitespace. It skips blank values, preserves meaningful punctuation/hashtags/mentions/emoji, and splits long text into sentence-aware chunks no larger than the configured maximum. It does not apply platform-specific parsing or remove duplicate source records.

Each completed run contains:

• cleaned_records.csv, segments.csv, and excluded_records.csv: normalized input records, model segments, and exclusions.
• generation.json and irrelevant_segments.csv: generated initial codes and irrelevant segments.
• assignments.csv, theme_summary.csv, theme_codes.csv, and theme_examples.csv: code-to-theme assignments, counts, and evidence.
• hierarchy.json, clustering/, run_manifest.json, and run_config.json: hierarchy, exact clustering outputs, run identity, and configuration.
• execution_attempts.json and stage checkpoint directories: resumable execution history and completed concurrent work.

Counts are code occurrences: one segment can receive multiple codes.

Run the test suite from the repository root:

uv run python -m unittest discover -s tests -t .