Skip to content

gh-102856: Python tokenizer implementation for PEP 701 #104323

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pablogsal merged 20 commits into from
May 21, 2023
Merged

gh-102856: Python tokenizer implementation for PEP 701 #104323

Changes from 1 commit
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
008f8e5
First iteration
mgmacias95 Apr 19, 2023
67a6ad6
Handle escaping {
mgmacias95 Apr 27, 2023
f58104d
nested expressions
mgmacias95 Apr 29, 2023
26102cc
Recursive expression tokenization
mgmacias95 May 2, 2023
a5f4b40
Remove intermediate token created for dev purposes
mgmacias95 May 2, 2023
598bab4
More improvements
mgmacias95 May 3, 2023
a0ed816
fix handling of } tokens
mgmacias95 May 7, 2023
90b4ab1
other tokenizer
pablogsal May 16, 2023
63ef1c1
Some progress
pablogsal May 17, 2023
6833b1a
Fix more bugs
pablogsal May 18, 2023
90da796
Fix more problems
pablogsal May 18, 2023
b5ccd94
Use IA to clean code
pablogsal May 18, 2023
b1c3b2a
Remove lel
pablogsal May 18, 2023
e941f12
Remove whitespace
pablogsal May 18, 2023
67a0239
Fix docs
mgmacias95 May 18, 2023
dcd221f
Moar tests and fix location error
pablogsal May 19, 2023
fd8b60a
Some cleanups
pablogsal May 19, 2023
f1a5090
pass the vacuum cleaner
pablogsal May 19, 2023
7fb58b0
Fix refleaks
mgmacias95 May 20, 2023
e1b5d35
📜🤖 Added by blurb_it.
blurb-it[bot] May 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
📜🤖 Added by blurb_it.
  • Loading branch information
@blurb-it
blurb-it[bot] authored May 20, 2023
commit e1b5d352c07994da3e3d85ab94086c1f1c068409
1 change: 1 addition & 0 deletions Misc/NEWS.d/next/Core and Builtins/2023-05-20-23-08-48.gh-issue-102856.Knv9WT.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Implement PEP 701 changes in the :mod:`tokenize` module. Patch by Marta Gómez Macías and Pablo Galindo Salgado