Amharic Tokenizer 🇪🇹

Amharic tokenizer with a GPT-style BPE-like pipeline over decomposed fidel. Implements: cleaning → fidel decomposition → BPE training/application → detokenization, with a Cython core for speed.

Training on a 16-million-word corpus: 12 hours → 42 seconds (~1,000× faster). Tokenization is also linear in text length instead of quadratic.

Same corpus, same 30,000-token vocabulary, same machine.

• train() previously went through every word in the corpus and copied all pair counts for every merge.
• Each distinct word is now stored once with its frequency, an index maps each pair to the words that contain it, and a heap tracks the most frequent pair. Each merge only updates the words that contain the merged pair.
• The learned merges, counts and IDs match the v0.2.6 algorithm on the benchmark corpora. Ties still go to the pair seen first in the corpus.
• With --verbose, progress is printed one line at a time with elapsed time.

• tokenize() and encode() previously applied merges to the whole text at once: every merge step scanned and rebuilt every word to find a single best pair, so time grew quadratically with text length.
• Merges never cross word boundaries (each word ends with <eow>), so each word is now tokenized independently, and results are cached per word so repeated words are not tokenized again.
• Merge lookups use integer symbol ids in a C++ hash table instead of building a new string for every pair, so the inner loop runs without Python objects.
• encode() now also accepts the token list returned by tokenize() (tok.encode(tokens)), so tokenizing and then encoding no longer tokenizes twice.
• For the same model, tokens and IDs are identical to v0.2.6.

Measured with amh_bpe_v0.2.6 on text from data/cleaned_data.txt, starting from an empty word cache each time. The first tokenize() call after load() or train() also builds the merge lookup table once (about 0.05 s).

• v0.2.7 ships an updated fidel map and a new pretrained model, amh_bpe_v0.2.7. The v0.2.6 model is still included, so existing code keeps working:

  from amharic_tokenizer import AmharicTokenizer
  
  tok = AmharicTokenizer.load("amh_bpe_v0.2.7")  # new model
  old = AmharicTokenizer.load("amh_bpe_v0.2.6")  # previous model

• Upgrading: token IDs from amh_bpe_v0.2.6 and amh_bpe_v0.2.7 are not interchangeable. If you stored IDs, keep decoding them with the model that produced them.

• src/ layout with a small Cython core (_bpe.pyx) behind a pure-Python, type-checked API. See docs/architecture.md.
• AmharicTokenizer.load() now also loads models from any file path (previously only bundled models could be loaded), and defaults to the latest bundled model.
• New CLI commands: clean, tokenize, encode, decode, models.
• Typed exceptions (ModelNotFoundError, InvalidModelError, TrainingError), logging instead of print, py.typed marker.
• Unit and integration test suites (pinned against recorded v0.2.7 outputs), CI on Linux/macOS/Windows and Python 3.9–3.13.

Speed on your own machine:

python benchmarks/benchmark.py                                    # small bundled sample
python benchmarks/benchmark.py data/cleaned_data.txt --num-merges 30000

• 30,000-token vocabulary trained on a larger, more diverse Amharic corpus.
• Pretrained models load by name: AmharicTokenizer.load("amh_bpe_v0.2.6").
• Full token ↔ id round trip (tokenize, encode, decode, detokenize).

pip install amharic-tokenizer
amh-tokenizer --help

From source (compiles the Cython extension):

python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

from amharic_tokenizer import AmharicTokenizer

tok = AmharicTokenizer.load()  # latest bundled model (amh_bpe_v0.2.7)

text = "ኢትዮጵያ ጥሩ ናት"
tokens = tok.tokenize(text)  # subword tokens; each word ends with <eow>
ids = tok.encode(text)  # or tok.encode(tokens) to skip re-tokenizing
assert tok.decode(ids) == text
assert tok.detokenize(tokens) == text

Characters outside the vocabulary (Latin letters, digits, most punctuation) are encoded as <unk> and dropped by decode().

# Optional: turn raw scraped text into unique, cleaned sentences
amh-tokenizer clean raw_amharic.txt cleaned_amharic.txt

# Train and save (.json is appended when missing)
amh-tokenizer train cleaned_amharic.txt models/amh_bpe \
  --num-merges 50000 --max-vocab-size 30000 --verbose --log-every 2000

from amharic_tokenizer import AmharicTokenizer, train_from_file

result = train_from_file("cleaned_amharic.txt", "models/amh_bpe", num_merges=50000)
tok = AmharicTokenizer.load(result.model_path)

# or, in memory
tok = AmharicTokenizer(num_merges=2000, max_vocab_size=5000)
tok.train(corpus_text, verbose=True, log_every=100)
tok.save("amh_bpe_model")

-m/--model accepts a file path or a bundled model name. Add -v for debug logs.

AmharicTokenizer(num_merges=50000, max_vocab_size=30000)

Other exports: train_from_file, list_pretrained_models, decompose, compose, AMHARIC_FIDEL_MAP, REVERSE_FIDEL_MAP, and the exceptions AmharicTokenizerError, ModelNotFoundError, InvalidModelError and TrainingError.

Upgrading from ≤ 0.2.6: amharic_tokenizer.fidel_map is now amharic_tokenizer.fidel, amharic_tokenizer.pipeline is now amharic_tokenizer.training (train_and_save is kept), and the private _vocabulary, _merge_rank_map, … attributes are replaced by the properties above. Load and save no longer print; use logging to see those messages.

src/amharic_tokenizer/   the package (see docs/architecture.md)
tests/unit/              fast, isolated module tests
tests/integration/       bundled models, training pipeline, CLI
tests/fixtures/          sample corpus + golden outputs
benchmarks/              speed measurements
tools/                   corpus crawler (not installed)
docs/                    architecture notes

pip install -e ".[dev]"
pytest                          # all tests
pytest tests/unit               # unit only
pytest -m integration           # integration only
ruff check . && ruff format --check .
mypy
tox                             # everything, on every installed Python

After editing _bpe.pyx, rebuild with pip install -e . (or python setup.py build_ext --inplace). See CONTRIBUTING.md.

• Longer, more diverse corpora and higher num_merges produce longer subwords.
• Training and tokenization work over decomposed fidel; detokenization recomposes the original Amharic characters.
• Tokenized words are cached per tokenizer (up to 200,000 distinct words); the cache is reset by train() and load().
• Token ids are model-specific: decode ids with the model that produced them.

• ModuleNotFoundError inside the repo: install in editable mode (pip install -e .).
• TestPyPI installs: resolve build dependencies from PyPI:

pip install -i https://test.pypi.org/simple/ \
    --extra-index-url https://pypi.org/simple amharic-tokenizer

This project is licensed under the MIT License – see the LICENSE file for details.