While working on GLM5.2 chat-templating for Basetenkenizer, found a small bug, thats now fixed with latest minijinja 2.22 (x.com/feilsystem/sta…) and will render the chat-template correctly. Will be included in future releases.
github.com/mitsuhiko/mini…
We shipped the worlds fastest tokenizer for KimiK3. Basetenkenizer now available on day 0 on Baseten and on pypi.
Article
Making Kimi K3 tokenization 18x faster for million-token agentic workloads
Today, we are releasing the Baseten Tokenizer (Basetenkenizer) to optimize tokenization for long input sequences, starting with Kimi K3.
For years, inference engineers have been able to disregard...
