Skip to content

PEP 3131 -- Supporting Non-ASCII Identifiers #607

Closed
Closed
PEP 3131 -- Supporting Non-ASCII Identifiers#607

Description

@holymonson
holymonson
opened on Aug 25, 2018

Python3 already support non-ASCII identifiers, pep-3131. But yapf fails to parse.

See example below, 中 is a non-ASCII token.

$ echo '中=1' | yapf
Traceback (most recent call last):
  File "/usr/local/miniconda3/lib/python3.7/site-packages/yapf/yapflib/pytree_utils.py", line 115, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/driver.py", line 104, in parse_string
    return self.parse_tokens(tokens, debug)
  File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/driver.py", line 72, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/parse.py", line 116, in addtoken
    ilabel = self.classify(type, value, context)
  File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/parse.py", line 172, in classify
    raise ParseError("bad token", type, value, context)
lib2to3.pgen2.parse.ParseError: bad token: type=58, value='中', context=('', (1, 0))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda3/lib/python3.7/site-packages/yapf/yapflib/yapf_api.py", line 129, in FormatCode
    tree = pytree_utils.ParseCodeToTree(unformatted_source)
  File "/usr/local/miniconda3/lib/python3.7/site-packages/yapf/yapflib/pytree_utils.py", line 121, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/driver.py", line 104, in parse_string
    return self.parse_tokens(tokens, debug)
  File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/driver.py", line 72, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/parse.py", line 116, in addtoken
    ilabel = self.classify(type, value, context)
  File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/parse.py", line 172, in classify
    raise ParseError("bad token", type, value, context)
lib2to3.pgen2.parse.ParseError: bad token: type=58, value='中', context=('', (1, 0))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda3/bin/yapf", line 10, in <module>
    sys.exit(run_main())
  File "/usr/local/miniconda3/lib/python3.7/site-packages/yapf/__init__.py", line 313, in run_main
    sys.exit(main(sys.argv))
  File "/usr/local/miniconda3/lib/python3.7/site-packages/yapf/__init__.py", line 183, in main
    verify=args.verify)
  File "/usr/local/miniconda3/lib/python3.7/site-packages/yapf/yapflib/yapf_api.py", line 131, in FormatCode
    raise parse.ParseError(filename + ': ' + e.message)
AttributeError: 'ParseError' object has no attribute 'message'```

Activity

  1. kamahen commented on Aug 25, 2018

    @kamahen
    kamahen
    on Aug 25, 2018
    Contributor

    Strange. Here's some code I wrote for testing lib2to3's encoding utility ... yapf handles it fine. So, maybe the problem is with a one-line program?

    import io
from lib2to3.pgen2 import tokenize
s='網目錦蛇=1'
sb = s.encode('utf8')
sb_f = io.BytesIO(sb)
print( tokenize.detect_encoding(sb_f.readline) )

    When run, this outputs:

    ('utf-8', [b'\xe7\xb6\xb2\xe7\x9b\xae\xe9\x8c\xa6\xe8\x9b\x87=1'])

    and yapf reformats the code to:

    import io
from lib2to3.pgen2 import tokenize
s = '網目錦蛇=1'
sb = s.encode('utf8')
sb_f = io.BytesIO(sb)
print(tokenize.detect_encoding(sb_f.readline))

  2. holymonson commented on Aug 25, 2018

    @holymonson
    holymonson
    on Aug 25, 2018
    Author

    maybe the problem is with a one-line program?

    No, this bug is found when parsing file, the one-line program is just a sample.

    Below example is clearer.

    from yapf.yapflib.pytree_utils import ParseCodeToTree

ParseCodeToTree('a = 1\n')
# Node(file_input, [Node(simple_stmt, [Node(expr_stmt, [Leaf(1, 'a'), Leaf(22, '='), Leaf(2, '1')]), Leaf(4, '\n')]), Leaf(0, '')])

ParseCodeToTree('中 = 1\n')
# Traceback (most recent call last):
#   File "/usr/local/miniconda3/lib/python3.7/site-packages/yapf/yapflib/pytree_utils.py", line 115, in ParseCodeToTree
#     tree = parser_driver.parse_string(code, debug=False)
#   File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/driver.py", line 104, in parse_string
#     return self.parse_tokens(tokens, debug)
#   File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/driver.py", line 72, in parse_tokens
#     if p.addtoken(type, value, (prefix, start)):
#   File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/parse.py", line 116, in addtoken
#     ilabel = self.classify(type, value, context)
#   File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/parse.py", line 172, in classify
#     raise ParseError("bad token", type, value, context)
# lib2to3.pgen2.parse.ParseError: bad token: type=58, value='中', context=('', (1, 0))
# 
# During handling of the above exception, another exception occurred:
# 
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
#   File "/usr/local/miniconda3/lib/python3.7/site-packages/yapf/yapflib/pytree_utils.py", line 121, in ParseCodeToTree
#     tree = parser_driver.parse_string(code, debug=False)
#   File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/driver.py", line 104, in parse_string
#     return self.parse_tokens(tokens, debug)
#   File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/driver.py", line 72, in parse_tokens
#     if p.addtoken(type, value, (prefix, start)):
#   File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/parse.py", line 116, in addtoken
#     ilabel = self.classify(type, value, context)
#   File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/parse.py", line 172, in classify
#     raise ParseError("bad token", type, value, context)
# lib2to3.pgen2.parse.ParseError: bad token: type=58, value='中', context=('', (1, 0))

  3. kamahen commented on Aug 25, 2018

    @kamahen
    kamahen
    on Aug 25, 2018
    Contributor

    Which version of Python are you using? Your sample program worked for me with Python 3.5, 3.6, 3.7 (I don't have anything older installed).

  4. holymonson commented on Aug 25, 2018

    @holymonson
    holymonson
    on Aug 25, 2018
    Author

    Which version of Python are you using?

    sys.version
# '3.7.0 (default, Jun 28 2018, 07:39:16) \n[Clang 4.0.1 (tags/RELEASE_401/final)]'
yapf.__version__
# '0.22.0'

  5. kamahen commented on Aug 25, 2018

    @kamahen
    kamahen
    on Aug 25, 2018
    Contributor

    I have the August 6 version of Python, but that shouldn't matter because I got similar results with 3.5 and 3.6.

    This is how I ran my test:

    $ cd src/yapf
$ git remote show origin
* remote origin
  Fetch URL: https://github.com/google/yapf.git
  Push  URL: https://github.com/google/yapf.git
  HEAD branch: master
  Remote branches:
    master                     tracked
    no-split-before-dict-value tracked
    style-config               tracked
    testrun-discovery          tracked
  Local branch configured for 'git pull':
    master merges with remote master
  Local ref configured for 'git push':
    master pushes to master (up to date)
$ python3.7 -m yapf /tmp/e.py
from yapf.yapflib.pytree_utils import ParseCodeToTree

ParseCodeToTree('a = 1\n')
# Node(file_input, [Node(simple_stmt, [Node(expr_stmt, [Leaf(1, 'a'), Leaf(22, '='), Leaf(2, '1')]), Leaf(4, '\n')]), Leaf(0, '')])

ParseCodeToTree('中 = 1\n')
$ find . -type f | xargs grep __version

./yapftests/main_test.py:      version = 'yapf {}\n'.format(yapf.__version__)
./setup.py:      version=yapf.__version__,
./yapf/__init__.py:__version__ = '0.22.0'
./yapf/__init__.py:    print('yapf {}'.format(__version__))

  6. holymonson commented on Aug 26, 2018

    @holymonson
    holymonson
    on Aug 26, 2018
    Author

    tokenize recognizes 'a' and 'a中' as NAME, but '中' as ERRORTOKEN.

    $ echo 'a = 1' | python3.7 -m lib2to3.pgen2.tokenize
1,0-1,1:	NAME	'a'
1,2-1,3:	OP	'='
1,4-1,5:	NUMBER	'1'
1,5-1,6:	NEWLINE	'\n'
2,0-2,0:	ENDMARKER	''

$ echo '中 = 1' | python3.7 -m lib2to3.pgen2.tokenize
1,0-1,1:	ERRORTOKEN	'中'
1,2-1,3:	OP	'='
1,4-1,5:	NUMBER	'1'
1,5-1,6:	NEWLINE	'\n'
2,0-2,0:	ENDMARKER	''

$ echo 'a中 = 1' | python3.7 -m lib2to3.pgen2.tokenize
1,0-1,2:	NAME	'a中'
1,3-1,4:	OP	'='
1,5-1,6:	NUMBER	'1'
1,6-1,7:	NEWLINE	'\n'
2,0-2,0:	ENDMARKER	''

    It seems should blame tokenize.py#L504 , only those token initialized with namechars = string.ascii_letters + '_' can be recognized as NAME, while Python 3 Interpreter dosen't restrict that.

                    elif initial in namechars:                 # ordinary name

    However, lib2to3.pgen2 seems a parser aiming for converting Python 2 code to Python 3, not parsing Python 3 code (I'm not sure). And it may retire when the End-Of-Life date of Python 2 (2020-01-01) comes.

    @kamahen , I don't know why your python works, is there a parser other than lib2to3.pgen2.tokenize?

  7. kamahen commented on Aug 26, 2018

    @kamahen
    kamahen
    on Aug 26, 2018
    Contributor

    @holymonson - I can reproduce your results.

    I agree -- it looks like a bug in lib2to3.pgen2.tokenize.

  8. holymonson commented on Aug 27, 2018

    @holymonson
    holymonson
    on Aug 27, 2018
    Author

    xfer python/cpython#8950

  9. holymonson
    closed this as completedon Apr 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions