Skip to content

bpo-33338: [lib2to3] Synchronize token.py and tokenize.py with stdlib #6572

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ambv wants to merge 3 commits into from
Closed

bpo-33338: [lib2to3] Synchronize token.py and tokenize.py with stdlib #6572

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
dafd047
[lib2to3] Synchronize token.py and tokenize.py with stdlib
ambv Apr 22, 2018
77ae1de
NEWS.d
ambv Apr 23, 2018
a6a3957
D'oh. Blurb hates bullet points.
ambv Apr 23, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 6 additions & 8 deletions Misc/NEWS.d/next/Library/2018-04-22-18-27-13.bpo-33338.185_om.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
lib2to3's tokenizer was synchronized with the built-in tokenizer, providing
many wins, including:

- the ``tokenize.tokenize()`` API is now mimicked in ``lib2to3``, including
automatic encoding detection and yielding ENCODING tokens; - Unicode
identifiers are now supported; - ELLIPSIS is its own token type now; -
Untokenizer improved with backports of 5e6db31, 9dc3a36, 5b8d2c3, e411b66,
and BPO-2495.

@merwok merwok Nov 22, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem was:

Warning, treated as error:
../build/NEWS:137:Bullet list ends without a blank line; unexpected unindent.

which I think means that this is the correct rst format:

- bla bla item one
  second line is indentend

- bla bla second item after blank line

lib2to3's tokenizer was synchronized with the built-in tokenizer. This
provides multiple wins. The ``tokenize.tokenize()`` API is now mimicked
in ``lib2to3``, including automatic encoding detection and yielding of
the ENCODING token. Unicode identifiers are now supported. ELLIPSIS is
its own token type now. Untokenizer improved with backports of 5e6db31,
9dc3a36, 5b8d2c3, e411b66, and BPO-2495.