Skip to content

gh-102856: Python tokenizer implementation for PEP 701 #104323

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pablogsal merged 20 commits into from
May 21, 2023
Merged

gh-102856: Python tokenizer implementation for PEP 701 #104323

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
008f8e5
First iteration
mgmacias95 Apr 19, 2023
67a6ad6
Handle escaping {
mgmacias95 Apr 27, 2023
f58104d
nested expressions
mgmacias95 Apr 29, 2023
26102cc
Recursive expression tokenization
mgmacias95 May 2, 2023
a5f4b40
Remove intermediate token created for dev purposes
mgmacias95 May 2, 2023
598bab4
More improvements
mgmacias95 May 3, 2023
a0ed816
fix handling of } tokens
mgmacias95 May 7, 2023
90b4ab1
other tokenizer
pablogsal May 16, 2023
63ef1c1
Some progress
pablogsal May 17, 2023
6833b1a
Fix more bugs
pablogsal May 18, 2023
90da796
Fix more problems
pablogsal May 18, 2023
b5ccd94
Use IA to clean code
pablogsal May 18, 2023
b1c3b2a
Remove lel
pablogsal May 18, 2023
e941f12
Remove whitespace
pablogsal May 18, 2023
67a0239
Fix docs
mgmacias95 May 18, 2023
dcd221f
Moar tests and fix location error
pablogsal May 19, 2023
fd8b60a
Some cleanups
pablogsal May 19, 2023
f1a5090
pass the vacuum cleaner
pablogsal May 19, 2023
7fb58b0
Fix refleaks
mgmacias95 May 20, 2023
e1b5d35
📜🤖 Added by blurb_it.
blurb-it[bot] May 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Moar tests and fix location error
  • Loading branch information
@pablogsal
pablogsal committed May 19, 2023
commit dcd221f566a38ed622149a2d0835a743a987ef1f
26 changes: 20 additions & 6 deletions Lib/test/test_tokenize.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -421,12 +421,6 @@ def test_string(self):
FSTRING_MIDDLE '}' (1, 9) (1, 10)
FSTRING_END '"' (1, 11) (1, 12)
""")
# TODO: I don't think is is correct now (ERRORTOKEN)
# self.check_tokenize('f"{1+1"', """\
# FSTRING_START 'f"' (1, 0) (1, 2)
# ERRORTOKEN '{' (1, 2) (1, 3)
# FSTRING_END '"' (1, 3) (1, 4)
# """)
self.check_tokenize('f"""{f\'\'\'{f\'{f"{1+1}"}\'}\'\'\'}"""', """\
FSTRING_START 'f\"""' (1, 0) (1, 4)
OP '{' (1, 4) (1, 5)
Expand All @@ -447,6 +441,26 @@ def test_string(self):
FSTRING_END "'''" (1, 24) (1, 27)
OP '}' (1, 27) (1, 28)
FSTRING_END '\"""' (1, 28) (1, 31)
""")
self.check_tokenize('f""" x\nstr(data, encoding={invalid!r})\n"""', """\
FSTRING_START 'f\"""' (1, 0) (1, 4)
FSTRING_MIDDLE ' x\\nstr(data, encoding=' (1, 4) (2, 19)
OP '{' (2, 19) (2, 20)
NAME 'invalid' (2, 20) (2, 27)
OP '!' (2, 27) (2, 28)
NAME 'r' (2, 28) (2, 29)
OP '}' (2, 29) (2, 30)
FSTRING_MIDDLE ')\\n' (2, 30) (3, 0)
FSTRING_END '\"""' (3, 0) (3, 3)
""")
self.check_tokenize('f"""123456789\nsomething{None}bad"""', """\
FSTRING_START 'f\"""' (1, 0) (1, 4)
FSTRING_MIDDLE '123456789\\nsomething' (1, 4) (2, 9)
OP '{' (2, 9) (2, 10)
NAME 'None' (2, 10) (2, 14)
OP '}' (2, 14) (2, 15)
FSTRING_MIDDLE 'bad' (2, 15) (2, 18)
FSTRING_END '\"""' (2, 18) (2, 21)
""")
self.check_tokenize('f"""abc"""', """\
FSTRING_START 'f\"""' (1, 0) (1, 4)
Expand Down
2 changes: 1 addition & 1 deletion Parser/tokenizer.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1126,7 +1126,7 @@ tok_underflow_interactive(struct tok_state *tok) {

static int
tok_underflow_file(struct tok_state *tok) {
if (tok->start == NULL) {
if (tok->start == NULL && !INSIDE_FSTRING(tok)) {
tok->cur = tok->inp = tok->buf;
}
if (tok->decoding_state == STATE_INIT) {
Expand Down