grouped-query-attention

several types of attention modules written in PyTorch for learning purposes

Image Captioning With MobileNet-LLaMA 3

Modular, step-by-step implementation of GPT & Modern LLMs from scratch in PyTorch. Featuring RoPE, RMSNorm, SwiGLU, GQA, KV-Cache, Unit Tests, and an Interactive Web Playground.

Grouped-Query Attention (GQA) is an optimization method for transformer attention mechanisms that divides query heads into groups, with each group sharing a single key and value head.

Transformer attention, worked through numerically; from self-attention and RoPE to KV cache, MQA, and GQA.

A 25M-parameter tool-calling LLM built completely from scratch in PyTorch — BPE tokenizer, LLaMA-style architecture (RoPE, GQA, SwiGLU, RMSNorm), pretraining, SFT, DPO, and quantized export. A hands-on curriculum for understanding how LLMs actually work.

Decoder-only Assamese LM built from scratch — RMSNorm/RoPE/GQA/SwiGLU, 74.8M params. Includes tokenizer training, data pipeline, training loop, evaluation, and attention visualization. Foundation model for NodiAI, a privacy-focused Assamese document-assistant for schools, clinics, and legal offices

Transformer Models for Humorous Text Generation. Fine-tuned on Russian jokes dataset with ALiBi, RoPE, GQA, and SwiGLU.Plus a custom Byte-level BPE tokenizer.

(Unofficial) building Hugging Face SmolLM-blazingly fast and small language model with PyTorch implementation of grouped query attention (GQA)

An experimental 5B-parameter Mixture-of-Experts language model trained from scratch.

Examine cost-effective methods for optimizing GQA configurations, comparing the performance with its counterparts like Multi-Head Attention (MHA) and Multi-Query Attention (MQA).

LLM 推理性能决策基线：TTFT/TPOT、KV Cache、吞吐与解码策略对照

From-scratch attention mechanisms in NumPy: scaled dot-product, self/cross, multi-head, causal/local, MQA/GQA, additive attention, and verified gradients.

Criando um modelo Transformer do zero com variações como Multi-Head Attention e Grouped Query Attention em livros de Machado de Assis.

Comparing transformer attention mechanisms (MHA, GQA, MQA, MLA) trained from scratch under identical conditions.

A single-file implementation of LLaMA 3, with support for jitting, KV caching and prompting

TinyBard: a 2.8M-parameter, character-level language model built from scratch in PyTorch with modern Llama-style components (RMSNorm, RoPE, grouped-query attention, SwiGLU), trained on Tiny Shakespeare.

Criando um modelo Transformer do zero com Positional Encoding / Posições treináveis, MultiHead Attention, KV Cache e Grouped Attention com alguns livros brasileiros.

Train decoder-only language models from scratch for Indic languages (starting with Hindi), with a LLaMA-2-style architecture (GQA, RoPE, SwiGLU), tokenizer, evaluation, and chat inference.