RLHF Annotation Studio

A web-based tool for collecting human preference data to train and fine-tune large language models via Reinforcement Learning from Human Feedback (RLHF).

RLHF Annotation Studio enables human annotators to compare pairs of LLM responses and express preferences — the core data collection step in the RLHF pipeline used to align models like GPT-4 and Claude.

• Side-by-side response comparison — annotators evaluate two responses to the same prompt
• Preference capture — select Response A, Response B, mark as tie, or skip
• Reasoning field — optional free-text rationale per annotation
• Live stats dashboard — tracks A wins, B wins, ties, and skip counts
• JSONL export — exports preference data in the standard format for reward model training
• Batch progress tracking — visual progress bar across annotation sessions

Each annotation exports as a single JSON line:

{
  "id": 1,
  "prompt": "Explain gradient descent to a beginner.",
  "preferred": "B",
  "reasoning": "Response B used a clearer analogy and was easier to follow.",
  "timestamp": "2025-04-08T10:32:00.000Z"
}

This format is compatible with standard RLHF reward model training pipelines (TRL, DeepSpeed, OpenRLHF).

• React 18 with TypeScript
• CSS (no UI library — custom design system)

git clone https://github.com/Amankumarsingh23/rlhf-annotation-studio
cd rlhf-annotation-studio
npm install
npm start

Open http://localhost:3000 in your browser.

1. Prompt display — each task shows a prompt that was sent to two different model versions
2. Response evaluation — the annotator reads both responses and selects the preferred one
3. Preference submission — the annotation (with optional reasoning) is recorded
4. Export — all annotations export as JSONL for downstream reward model training

Human Annotations (this tool)
        ↓
  Reward Model Training
        ↓
  PPO Fine-tuning of LLM
        ↓
  Aligned Language Model

MIT