Skip to content

Commit 84a770c

Browse files
youknowoneyouknowone
committed
Add _tokenize module and update tokenize.py from CPython 3.14
Port _tokenize.TokenizerIter from PR RustPython#6240 (ShaharNaveh), adapted to current codebase. Uses ruff_python_parser for tokenization. Update Lib/tokenize.py from cpython/Lib/tokenize.py.
1 parent 04cf5da commit 84a770c

5 files changed

Lines changed: 525 additions & 221 deletions

File tree

‎Cargo.lock‎

Lines changed: 4 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)