OdiaGPT (ଓଡ଼ିଆଜିପିଟି)

Watch the complete visual architecture breakdown, tokenization benchmarking against LLaMA-3, and empirical training dynamics on YouTube:

OdiaGPT is an educational, production-clean research project that implements and trains an autoregressive decoder-only Transformer specifically for the Odia (ଓଡ଼ିଆ) language completely from scratch.

Unlike common transfer-learning workflows, OdiaGPT adheres strictly to a zero-pretrained-dependency policy:

• ❌ No pretrained weights: Zero GPT-2, Llama, Mistral, or Indic foundation model checkpoints.
• ❌ No off-the-shelf tokenizers: No Llama, IndicBERT, or multilingual tokenizers.
• ❌ No ready-made model wrappers: No GPT2LMHeadModel or AutoModelForCausalLM during model construction.
• ✅ 100% PyTorch primitives: Every tensor operation, rotary embedding, normalization layer, and attention mechanism is written from the ground up.
• ✅ From-scratch Indic Tokenizer: An 8,192-vocabulary SentencePiece BPE tokenizer trained exclusively on cleaned Odia text with byte-fallback and 100% character coverage.

The flagship model, OdiaGPT-30M (31,465,984 parameters), was trained on Google Colab (NVIDIA Tesla T4) over 29.5 million tokens from a filtered, NFC-normalized subset of the AI4Bharat Sangraha verified Odia partition.

OdiaGPT utilizes state-of-the-art decoder-only Transformer design principles:

The model is natively exported to Hugging Face format and published to the Hugging Face Hub under CodeHima/OdiaGPT-30M.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "CodeHima/OdiaGPT-30M"

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
    device_map="auto"
)

# Prompt in Odia
prompt = "ଓଡ଼ିଶାର ଲୋକମାନେ"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# Generate with nucleus sampling and repetition penalty
outputs = model.generate(
    **inputs,
    max_new_tokens=64,
    temperature=0.8,
    top_k=50,
    top_p=0.9,
    repetition_penalty=1.15,
    do_sample=True,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

An interactive web interface with dual-model selection (30M vs Nano), generation parameters, and one-click Odia presets is included:

• Live Hugging Face Space: huggingface.co/spaces/CodeHima/OdiaGPT-Web
• Live Colab Session: 7165f80b8a0ec54d28.gradio.live

To run the Gradio interface locally:

git clone https://github.com/HimanshuMohanty-Git24/OdiaGPT.git
cd OdiaGPT
pip install -r requirements.txt
python app.py

The project followed a rigorous 22-stage curriculum:

Smoke Memorization (33 tokens) ──> Nano Experiment (5.5M params) ──> OdiaGPT-30M (31.5M params)
     Loss: 8.33 -> 0.43                   Val PPL: 373.30                      Val PPL: 106.09

• Throughput: ~35,202 tokens/sec on a single NVIDIA Tesla T4 GPU.
• VRAM Utilization: Peak allocated 3.53 GiB (only 24.2% of 14.56 GiB T4 memory; zero OOMs).
• Optimization: AdamW ($\beta_1=0.9, \beta_2=0.95$, weight decay 0.1, grad clip 1.0) with linear warmup (100 steps) and cosine learning rate decay ($3 \times 10^{-4} \to 3 \times 10^{-5}$).

Samples generated from the final OdiaGPT-30M checkpoint (step_0001800.pt) under Nucleus Sampling ($T=0.8, \text{top-k}=50, \text{top-p}=0.9, \text{rep-penalty}=1.15$):

ଓଡ଼ିଶାର ଲୋକମାନେ ବିରିୟାରେ ସାନ୍ଦ୍ରାର ଅଭାବକୁ ନେଇ ନିଜକୁ ନିଜେ ଅନେକ କଥା କହୁଥିବା କଥା । ଆମେ ସମସ୍ତେ ଜାଣନ୍ତି ଯେ ଏହି ମହାଭାରତ ତଥା ଦେଶର ବିଭିନ୍ନ ରାଜ୍ୟରୁ ଆସୁଥିବା ଓଡ଼ିଆଙ୍କ ପାଇଁ ଏକ ପ୍ରକାର ପ୍ରୟାସ । ଏହି ସମୟରେ ଆମେ ସମସ୍ତେ ସମସ୍ତେ ଏହି ଅବସରରେ ରହି ଏହିସବୁ କାର୍ଯ୍ୟକ୍ରମ ବିଷୟରେ କହିବାକୁ ଯାଉଛୁ । ଆମେ ମଧ୍ୟ ଏହି ସଭାରେ ଅଂଶଗ୍ରହଣ କରିବା ପାଇଁ ଏକ ନୂଆ ପ୍ରସ୍ତୁତି
Linguistic Analysis: Learned accurate compound clause formation, correct postpositions (ପାଇଁ, ବିଷୟରେ), plural noun markers (ଙ୍କ), and topical Odia vocabulary (ଓଡ଼ିଆଙ୍କ, ରାଜ୍ୟରୁ, କାର୍ଯ୍ୟକ୍ରମ).

ଆଜି ସକାଳେ ସକାଳେ ଦୁଇ ଝିଅ ସହ ଆସି ଶୋଇପଡ଼ିଲା । ସେହି ସମୟରେ ବାପା ଘର ଲୋକ କୁ ଖବର ଦେଇ କହିଥିଲେ ଯେ, "ତେବେ ମାଆ, ତମେ ଆଉ କିଛି କହିଦେବୁ । ମୁଁ ଏ କଥା କିଛି ଶୁଣିନାହିଁ । କିନ୍ତୁ ବାପା, ମା'ଙ୍କୁ ନେଇ ମା'ର ମା'ର ପୁଅ ଝିଅ । ଏ କଥା କହି ସେ ଆଉ କାହା କଥା ଜାଣିନଥିଲା ।
Linguistic Analysis: Demonstrates natural narrative pacing, correct past tense verb agreement (ଆସି ଶୋଇପଡ଼ିଲା, କହିଥିଲେ, ଜାଣିନଥିଲା), dialogue quotation marks, and familial nouns.

ଓଡ଼ିଆ ଭାଷାର ଅନେକ ଦୃଶ୍ୟ, ଚିତ୍ର-ଅର୍ଣ୍ଣିମ, ନୂଆବୋହୂକ ଓ ଲୋକନାଥଙ୍କ ଭଳି କିଛି ନୂଆ କାହାଣୀକୁ ନେଇ ଅନେକ ପ୍ରଶ୍ନବାଚୀ ସୃଷ୍ଟି ହୋଇଛି । ତେଣୁ ଏହି ଦୁଇ ଚରିତ୍ରର କଥା ଜାଣିଲେ ଆଶ୍ଚର୍ଯ୍ୟ ହୋଇଯିବେ । ପ୍ରଧାନମନ୍ତ୍ରୀଙ୍କ ଚିଠିରେ ପ୍ରକାଶ ପାଇବା ପରେ ଅନେକ ଘଟଣା ସାମ୍ନାକୁ ଆସିଛି ।
Linguistic Analysis: Formal, journalistic Odia tone with complex nominal morphology and cohesive discourse markers (ତେଣୁ, ଯେଉଁଥିରେ).

OdiaGPT/
├── configs/               # Reproducible experiment configurations
│   ├── smoke.yaml         # 2-layer smoke config
│   ├── nano.yaml          # 4-layer 5.5M config
│   └── odia30m.yaml       # 8-layer 31.5M flagship config
├── src/odiagpt/           # Core library
│   ├── model.py           # Transformer architecture (RMSNorm, SwiGLU, OdiaGPT)
│   ├── attention.py       # Causal multi-head self-attention with SDPA
│   ├── rope.py            # Rotary Position Embeddings
│   ├── tokenizer.py       # SentencePiece training and inference wrapper
│   ├── dataset.py         # Memory-mapped packed uint16 binary dataset loader
│   ├── generation.py      # Autoregressive generation (greedy, top-k, top-p)
│   ├── checkpoint.py      # Stateful optimizer, scheduler, and RNG checkpointing
│   ├── cleaning.py        # Conservative Odia script & boilerplate filters
│   └── unicode_utils.py   # NFC normalization and diacritic preservation
├── scripts/               # Training, benchmarking, and paper export scripts
│   ├── generate_paper_figures.py # 300-DPI publication figure generator
│   ├── build_paper_pdf.py # Headless Chromium PDF paper compiler
│   ├── train_pilot.py     # OdiaGPT-30M training execution
│   ├── generate.py        # Benchmark generation runner
│   ├── upload_to_hf.py    # Hugging Face Hub uploader
│   └── tiny_overfit.py    # Memorization verification test
├── tests/                 # Unit test suite
│   ├── test_cleaning.py   # Unicode normalization & filtering tests (13 tests)
│   ├── test_model.py      # Causality, shapes, gradients, and RoPE tests
│   └── test_sdpa.py       # FlashAttention numerical equivalency tests
├── reports/               # Technical documentation & academic paper
│   ├── research_paper.pdf # Complete 7-page publication-grade PDF paper
│   ├── research_paper.tex # Full compile-ready LaTeX source
│   ├── research_paper.md  # Markdown edition with KaTeX & mermaid
│   ├── figures/           # 300-DPI vector & raster publication diagrams
│   │   ├── fig1_data_pipeline.png / .pdf
│   │   ├── fig2_architecture.png / .pdf
│   │   ├── fig3_fertility.png / .pdf
│   │   └── fig4_training_curves.png / .pdf
│   ├── main_training_report.md
│   ├── evaluation_report.md
│   ├── final_report.md
│   └── data_quality_report.md
├── tokenizer/             # Trained 8K and 4K SentencePiece .model/.vocab files
├── app.py                 # Interactive Gradio application
├── requirements.txt       # Project dependencies
├── LICENSE                # MIT License
└── README.md              # Project documentation

Run the test suite to verify model causality, normalization, and attention implementation:

python -m unittest discover -s tests -v

• Base Foundation Model: OdiaGPT is a pure autoregressive next-token predictor. It is not instruction-tuned or RLHF aligned, and will not reliably answer questions in a conversational assistant format without fine-tuning.
• Factual Knowledge: Generations reflect statistical language patterns from web data; they should not be relied upon for factual or historical queries.
• Community Use: Provided freely under the MIT License for educational and linguistic research into low-resource Indic language models.

If you use OdiaGPT in your research or project, please cite:

@article{mohanty2026odiagpt,
  title     = {OdiaGPT: Pretraining an Indic Decoder-Only Transformer from Scratch for Low-Resource Odia Language Modeling},
  author    = {Mohanty, Himanshu},
  journal   = {arXiv preprint / GitHub Research Report},
  year      = {2026},
  url       = {https://github.com/HimanshuMohanty-Git24/OdiaGPT}
}

• Source Code: MIT License © 2026 Himanshu Mohanty
• Corpus Attribution: Trained on subsets of AI4Bharat Sangraha (verified/ori).
• Creator: Himanshu Mohanty (CodeHima)