Skip to content

bpo-30406: Make async and await proper keywords #1669

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
1st1 merged 26 commits into from
Oct 6, 2017
Merged

bpo-30406: Make async and await proper keywords #1669

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
abbd401
make async and await real keywords
JelleZijlstra Apr 16, 2017
8e16e6b
remove asyncio test that is now a SyntaxError
JelleZijlstra Apr 16, 2017
81dae40
update syntax tests in test_coroutines
JelleZijlstra Apr 16, 2017
9e65b47
regenerate symbol.py
JelleZijlstra Apr 16, 2017
f850135
fix tokenize.py
JelleZijlstra Apr 16, 2017
f95c992
fewer nested lists allowed
JelleZijlstra Apr 16, 2017
5c8bddc
regenerate keyword.py
JelleZijlstra Apr 16, 2017
ac04cff
fix lib2to3
JelleZijlstra Apr 16, 2017
8af009f
add some docs
JelleZijlstra Apr 22, 2017
4289253
fix test_parser.py
JelleZijlstra May 19, 2017
cf2f467
attempt to get pydoc to work
JelleZijlstra May 19, 2017
c824937
add blurb
JelleZijlstra Jul 21, 2017
ca85605
run make patchcheck to fix whitespace
JelleZijlstra Jul 21, 2017
505a66d
remove obsolete code from forbidden_name
JelleZijlstra Jul 22, 2017
57fb8ed
attempt to make it work without await_expr
JelleZijlstra Jul 22, 2017
e987d50
undo async_for-related changes
JelleZijlstra Jul 22, 2017
dacf3e6
minor CR fixes
JelleZijlstra Jul 22, 2017
d31627a
closer to yuri's code
JelleZijlstra Sep 27, 2017
ef28dea
undo unnecessary change
JelleZijlstra Aug 27, 2017
499aa40
fix whitespace (make patchcheck)
JelleZijlstra Sep 27, 2017
7c664e3
fix test_symbol
JelleZijlstra Sep 27, 2017
5659c8d
use a separate sync_comp_for (suggested by Yury)
JelleZijlstra Sep 30, 2017
2413e3e
regenerate symbol.py
JelleZijlstra Sep 30, 2017
cdbc7d5
simplify lib2to3 changes
JelleZijlstra Sep 30, 2017
38ba810
remove "again" from wording intoken.rst
JelleZijlstra Oct 5, 2017
78ddbde
fix lib2to3
JelleZijlstra Oct 6, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix lib2to3
  • Loading branch information
@JelleZijlstra
JelleZijlstra committed Oct 6, 2017
commit ac04cffb41b5253a3f18c2fc41b42237f9d4d07c
25 changes: 14 additions & 11 deletions Lib/lib2to3/Grammar.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ eval_input: testlist NEWLINE* ENDMARKER
decorator: '@' dotted_name [ '(' [arglist] ')' ] NEWLINE
decorators: decorator+
decorated: decorators (classdef | funcdef | async_funcdef)
async_funcdef: ASYNC funcdef
async_funcdef: 'async' funcdef
funcdef: 'def' NAME parameters ['->' test] ':' suite
parameters: '(' [typedargslist] ')'
typedargslist: ((tfpdef ['=' test] ',')*
Expand Down Expand Up @@ -85,7 +85,7 @@ exec_stmt: 'exec' expr ['in' test [',' test]]
assert_stmt: 'assert' test [',' test]

compound_stmt: if_stmt | while_stmt | for_stmt | try_stmt | with_stmt | funcdef | classdef | decorated | async_stmt
async_stmt: ASYNC (funcdef | with_stmt | for_stmt)
async_stmt: 'async' (funcdef | with_stmt | for_stmt)
if_stmt: 'if' test ':' suite ('elif' test ':' suite)* ['else' ':' suite]
while_stmt: 'while' test ':' suite ['else' ':' suite]
for_stmt: 'for' exprlist 'in' testlist ':' suite ['else' ':' suite]
Expand Down Expand Up @@ -123,15 +123,16 @@ and_expr: shift_expr ('&' shift_expr)*
shift_expr: arith_expr (('<<'|'>>') arith_expr)*
arith_expr: term (('+'|'-') term)*
term: factor (('*'|'@'|'/'|'%'|'//') factor)*
factor: ('+'|'-'|'~') factor | power
power: [AWAIT] atom trailer* ['**' factor]
factor: ('+'|'-'|'~') factor | await_expr
await_expr: 'await' power | power
power: atom trailer* ['**' factor]
atom: ('(' [yield_expr|testlist_gexp] ')' |
'[' [listmaker] ']' |
'{' [dictsetmaker] '}' |
'`' testlist1 '`' |
NAME | NUMBER | STRING+ | '.' '.' '.')
listmaker: (test|star_expr) ( old_comp_for | (',' (test|star_expr))* [','] )
testlist_gexp: (test|star_expr) ( old_comp_for | (',' (test|star_expr))* [','] )
listmaker: (test|star_expr) ( old_comp_for | comp_async_for | (',' (test|star_expr))* [','] )
testlist_gexp: (test|star_expr) ( old_comp_for | comp_async_for | (',' (test|star_expr))* [','] )
lambdef: 'lambda' [varargslist] ':' test
trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME
subscriptlist: subscript (',' subscript)* [',']
Expand All @@ -140,9 +141,9 @@ sliceop: ':' [test]
exprlist: (expr|star_expr) (',' (expr|star_expr))* [',']
testlist: test (',' test)* [',']
dictsetmaker: ( ((test ':' test | '**' expr)
(comp_for | (',' (test ':' test | '**' expr))* [','])) |
(comp_for | comp_async_for | (',' (test ':' test | '**' expr))* [','])) |
((test | star_expr)
(comp_for | (',' (test | star_expr))* [','])) )
(comp_for | comp_async_for | (',' (test | star_expr))* [','])) )

classdef: 'class' NAME ['(' [arglist] ')'] ':' suite

Expand All @@ -155,13 +156,15 @@ arglist: argument (',' argument)* [',']
# Illegal combinations and orderings are blocked in ast.c:
# multiple (test comp_for) arguments are blocked; keyword unpackings
# that precede iterable unpackings are blocked; etc.
argument: ( test [comp_for] |
argument: ( test [comp_for_or_async] |
test '=' test |
'**' expr |
star_expr )

comp_iter: comp_for | comp_if
comp_for: [ASYNC] 'for' exprlist 'in' or_test [comp_iter]
comp_for_or_async: comp_async_for | comp_for
comp_iter: comp_async_for | comp_for | comp_if
comp_async_for: 'async' 'for' exprlist 'in' testlist_safe [comp_iter]
comp_for: 'for' exprlist 'in' testlist_safe [comp_iter]
comp_if: 'if' old_test [comp_iter]

@1st1 1st1 Jul 21, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding to the point raised for changes in Grammar: when you change/rename/add new productions here, you might be breaking some code that uses lib2to3 to define custom fixters. Again, I'd avoid any changes.


# As noted above, testlist_safe extends the syntax allowed in list
Expand Down
6 changes: 2 additions & 4 deletions Lib/lib2to3/pgen2/token.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,10 +62,8 @@
COMMENT = 53
NL = 54
RARROW = 55
AWAIT = 56
ASYNC = 57
ERRORTOKEN = 58
N_TOKENS = 59
ERRORTOKEN = 56
N_TOKENS = 57
NT_OFFSET = 256
#--end constants--

Expand Down
76 changes: 2 additions & 74 deletions Lib/lib2to3/pgen2/tokenize.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,7 @@ def compat(self, token, iterable):
for tok in iterable:
toknum, tokval = tok[:2]

if toknum in (NAME, NUMBER, ASYNC, AWAIT):
if toknum in (NAME, NUMBER):
tokval += ' '

if toknum == INDENT:
Expand Down Expand Up @@ -380,12 +380,6 @@ def generate_tokens(readline):
contline = None
indents = [0]

# 'stashed' and 'async_*' are used for async/await parsing
stashed = None
async_def = False
async_def_indent = 0
async_def_nl = False

while 1: # loop over lines in stream
try:
line = readline()
Expand Down Expand Up @@ -426,10 +420,6 @@ def generate_tokens(readline):
pos = pos + 1
if pos == max: break

if stashed:
yield stashed
stashed = None

if line[pos] in '#\r\n': # skip comments or blank lines
if line[pos] == '#':
comment_token = line[pos:].rstrip('\r\n')
Expand All @@ -453,18 +443,8 @@ def generate_tokens(readline):
("<tokenize>", lnum, pos, line))
indents = indents[:-1]

if async_def and async_def_indent >= indents[-1]:
async_def = False
async_def_nl = False
async_def_indent = 0

yield (DEDENT, '', (lnum, pos), (lnum, pos), line)

if async_def and async_def_nl and async_def_indent >= indents[-1]:
async_def = False
async_def_nl = False
async_def_indent = 0

else: # continued statement
if not line:
raise TokenError("EOF in multi-line statement", (lnum, 0))
Expand All @@ -484,28 +464,17 @@ def generate_tokens(readline):
newline = NEWLINE
if parenlev > 0:
newline = NL
elif async_def:
async_def_nl = True
if stashed:
yield stashed
stashed = None
yield (newline, token, spos, epos, line)

elif initial == '#':
assert not token.endswith("\n")
if stashed:
yield stashed
stashed = None
yield (COMMENT, token, spos, epos, line)
elif token in triple_quoted:
endprog = endprogs[token]
endmatch = endprog.match(line, pos)
if endmatch: # all on one line
pos = endmatch.end(0)
token = line[start:pos]
if stashed:
yield stashed
stashed = None
yield (STRING, token, spos, (lnum, pos), line)
else:
strstart = (lnum, start) # multiple lines
Expand All @@ -523,63 +492,22 @@ def generate_tokens(readline):
contline = line
break
else: # ordinary string
if stashed:
yield stashed
stashed = None
yield (STRING, token, spos, epos, line)
elif initial in namechars: # ordinary name
if token in ('async', 'await'):
if async_def:
yield (ASYNC if token == 'async' else AWAIT,
token, spos, epos, line)
continue

tok = (NAME, token, spos, epos, line)
if token == 'async' and not stashed:
stashed = tok
continue

if token == 'def':
if (stashed
and stashed[0] == NAME
and stashed[1] == 'async'):

async_def = True
async_def_indent = indents[-1]

yield (ASYNC, stashed[1],
stashed[2], stashed[3],
stashed[4])
stashed = None

if stashed:
yield stashed
stashed = None

yield tok
yield (NAME, token, spos, epos, line)
elif initial == '\\': # continued stmt
# This yield is new; needed for better idempotency:
if stashed:
yield stashed
stashed = None
yield (NL, token, spos, (lnum, pos), line)
continued = 1
else:
if initial in '([{': parenlev = parenlev + 1
elif initial in ')]}': parenlev = parenlev - 1
if stashed:
yield stashed
stashed = None
yield (OP, token, spos, epos, line)
else:
yield (ERRORTOKEN, line[pos],
(lnum, pos), (lnum, pos+1), line)
pos = pos + 1

if stashed:
yield stashed
stashed = None

for indent in indents[1:]: # pop remaining indent levels
yield (DEDENT, '', (lnum, 0), (lnum, 0), '')
yield (ENDMARKER, '', (lnum, 0), (lnum, 0), '')
Expand Down
26 changes: 15 additions & 11 deletions Lib/lib2to3/tests/test_parser.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,34 +167,34 @@ def foo(): pass
async def foo(): await x
""")

self.invalid_syntax("await x")
self.invalid_syntax("""def foo():
await x""")
self.validate("await x")
self.validate("""def foo():
await x""")

self.invalid_syntax("""def foo():
self.validate("""def foo():
def foo(): pass
async def foo(): pass
await x
""")

def test_async_var(self):
self.validate("""async = 1""")
self.validate("""await = 1""")
self.validate("""def async(): pass""")
self.invalid_syntax("""async = 1""")
self.invalid_syntax("""await = 1""")
self.invalid_syntax("""def async(): pass""")

def test_async_with(self):
self.validate("""async def foo():
async for a in b: pass""")

self.invalid_syntax("""def foo():
async for a in b: pass""")
self.validate("""def foo():
async for a in b: pass""")

@vstinner vstinner May 20, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wait... this one looks like a bug.

@JelleZijlstra JelleZijlstra May 20, 2017 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think lib2to3's parser in general doesn't do semantic checking of the sort that would be required to reject this. For example, it also doesn't fail on putting nonlocal or yield from outside a function, or yield from inside an async generator.


def test_async_for(self):
self.validate("""async def foo():
async with a: pass""")

self.invalid_syntax("""def foo():
async with a: pass""")
self.validate("""def foo():
async with a: pass""")

@vstinner vstinner May 20, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto (also a bug)

@JelleZijlstra JelleZijlstra May 20, 2017

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, this is also lib2to3.



class TestRaiseChanges(GrammarTest):
Expand Down Expand Up @@ -477,3 +477,7 @@ def diff(fn, result):
os.remove("@")
except OSError:
pass


if __name__ == '__main__':
unittest.main()