RajNLP-50K: Rajasthani-Hindi Code-Switched NLP Corpus

The first open, annotated Rajasthani-Hindi code-switched NLP corpus — 50,000 sentences from Twitter/X and ShareChat, annotated for sentiment, named entity recognition (NER), and toxicity detection. Includes fine-tuned MuRIL models that outperform GPT-4o on all three tasks.

• Overview
• Dataset
• Models
• Installation
• Quick Start
• Running the Full Pipeline
• Training Models
• Evaluation
• Project Structure
• Citation
• License

Rajasthani-Hindi code-switching is pervasive on Indian social media but severely under-resourced in NLP. RajNLP-50K addresses this gap by providing:

• 50,000 annotated sentences from Twitter/X and ShareChat
• Three annotation layers: sentiment (3-class), NER (PER/LOC/ORG), toxicity (4-category multi-label)
• Token-level language ID labels (Rajasthani / Hindi / English / Transliterated)
• Fine-tuned MuRIL models for all three downstream tasks
• The first caste-based toxicity classifier for Rajasthani text

from datasets import load_dataset
ds = load_dataset("eeshsaxena/rajnlp-50k")

from models.muril_sentiment_classifier import MuRILSentimentClassifier

clf = MuRILSentimentClassifier()
clf.load("checkpoints/sentiment")
result = clf.predict("म्हारो राजस्थान घणो सुंदर है")
print(result.label, result.confidence)

• Python 3.10+
• CUDA-capable GPU (for training; inference works on CPU)

# Clone the repository
git clone https://github.com/eeshsaxena/rajnlp-50k.git
cd rajnlp-50k

# Create a virtual environment
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Copy environment template and fill in your API keys
cp .env.example .env

Create a .env file with:

TWITTER_BEARER_TOKEN=your_twitter_bearer_token
HF_TOKEN=your_huggingface_token

python run_pipeline.py --dry-run --seed 42 --output-dir output/test_run

This runs all 4 phases on a 100-sentence fixture and produces:

• output/test_run/corpus.jsonl — serialized corpus
• output/test_run/corpus.parquet — Parquet format
• output/test_run/pipeline.log — structured experiment log

pytest tests/ -v

Twitter/X (requires Academic API access):

export TWITTER_BEARER_TOKEN=your_token
python -c "
from corpus_builder.twitter_collector import TwitterCollector
import os
collector = TwitterCollector(bearer_token=os.environ['TWITTER_BEARER_TOKEN'])
sentences = collector.collect_twitter(
    query_terms=['rajasthan', 'राजस्थान', 'gehlot', 'vasundhara', '#rajasthan'],
    max_results=100000
)
print(f'Collected {len(sentences)} sentences')
"

ShareChat (requires Chrome + ChromeDriver):

pip install selenium webdriver-manager
python -c "
from corpus_builder.sharechat_collector import ShareChatCollector
urls = open('sharechat_urls.txt').read().splitlines()
collector = ShareChatCollector()
sentences = collector.collect_sharechat(urls)
print(f'Collected {len(sentences)} sentences')
"

1. Start Label Studio: label-studio start
2. Import project configs from annotator_tool/label_studio_configs/
3. Import sentences and assign to annotators
4. Export annotations and run IAA:

python -c "
from annotator_tool.iaa import compute_batch_iaa
# Load your exported annotations here
"

python train_all.py --seed 42 --data-dir output/annotated

python run_pipeline.py \
  --seed 42 \
  --output-dir output/run_001 \
  --log-level INFO

from models.muril_sentiment_classifier import MuRILSentimentClassifier
from corpus_builder.serialization import deserialize

# Load annotated data
sentences = deserialize("output/corpus.jsonl", fmt="jsonl")
train = [s for s in sentences if s.split == "train"]
val = [s for s in sentences if s.split == "validation"]

clf = MuRILSentimentClassifier()
log = clf.train(train, val, seed=42, max_epochs=10)
clf.save("checkpoints/sentiment")
print(f"Best F1: {log.best_f1:.4f}")

from models.muril_ner_tagger import MuRILNERTagger

tagger = MuRILNERTagger()
log = tagger.train(train, val, seed=42, max_epochs=5)
tagger.save("checkpoints/ner")

from models.muril_toxicity_classifier import MuRILToxicityClassifier

clf = MuRILToxicityClassifier()
log = clf.train(train, val, seed=42, max_epochs=10)
clf.save("checkpoints/toxicity")

python -c "
from models.muril_sentiment_classifier import MuRILSentimentClassifier
from corpus_builder.serialization import deserialize

sentences = deserialize('output/corpus.jsonl', fmt='jsonl')
test = [s for s in sentences if s.split == 'test']

clf = MuRILSentimentClassifier()
clf.load('checkpoints/sentiment')
metrics = clf.evaluate(test)
print(f'Sentiment macro-F1: {metrics.macro_f1:.4f}')
"

rajnlp-50k/
├── corpus_builder/          # Data collection, filtering, deduplication, serialization
│   ├── twitter_collector.py
│   ├── sharechat_collector.py
│   ├── filter_dedup.py
│   ├── sampling.py
│   ├── serialization.py
│   ├── span_validation.py
│   └── rajasthani_lexicon_full.txt
├── annotator_tool/          # Label Studio configs, IAA, majority vote, welfare
│   ├── label_studio_configs/
│   ├── iaa.py
│   ├── majority_vote.py
│   ├── export_converter.py
│   └── welfare.py
├── language_id/             # Token-level language boundary detector
│   ├── tagger.py
│   └── train.py
├── models/                  # Classifier implementations
│   ├── muril_sentiment_classifier.py   # Real MuRIL fine-tuning
│   ├── muril_ner_tagger.py             # Real MuRIL fine-tuning
│   ├── muril_toxicity_classifier.py    # Real MuRIL fine-tuning
│   ├── sentiment_classifier.py         # Heuristic stub (for testing)
│   ├── ner_tagger.py                   # Heuristic stub (for testing)
│   ├── toxicity_classifier.py          # Heuristic stub (for testing)
│   ├── reproducibility.py
│   └── data_models.py
├── evaluation/              # Baseline evaluators, comparison table, platform split
├── release/                 # HuggingFace publishing
├── tests/                   # 556 tests (pytest + hypothesis)
├── docs/                    # Annotation guidelines, IRB application, paper
├── run_pipeline.py          # Main pipeline orchestrator
├── requirements.txt
└── README.md

See docs/annotation_guidelines.md for:

• Labeling rules for all three tasks
• Worked examples with edge cases
• IAA thresholds and adjudication procedure
• Annotator compensation and welfare policies

This project involves annotation of toxic content including caste-based slurs. All annotators:

• Received a written content warning before starting
• Had access to an opt-out mechanism at any time
• Were limited to 2 hours/day of toxicity annotation
• Were compensated at ₹150–200/hour

IRB approval was obtained before annotation began. See docs/irb_application.md.

If you use RajNLP-50K in your research, please cite:

@dataset{rajnlp50k2024,
  title     = {{RajNLP-50K}: A Rajasthani-Hindi Code-Switched {NLP} Corpus},
  author    = {Saxena, Eesh},
  year      = {2024},
  publisher = {HuggingFace},
  url       = {https://huggingface.co/datasets/eeshsaxena/rajnlp-50k}
}

• Dataset: CC BY 4.0
• Code: Apache 2.0

Eesh Saxena — GitHub