Skip to content

PEP 617: new PEG-based parser #84514

Closed
Closed
PEP 617: new PEG-based parser#84514
Labels
3.9 (EOL)end of lifetype-featureA feature request or enhancement

Description

@gvanrossum
gvanrossum
opened on Apr 19, 2020
BPO 40334
Nosy @gvanrossum, @rhettinger, @ned-deily, @serhiy-storchaka, @Carreau, @asottile, @lysnikolaou, @pablogsal, @miss-islington, @isidentical, @hauntsaninja
PRs
  • bpo-40334: PEP 617: New PEG parser for CPython #19503
  • bpo-40334: Only run the CI with the new parser #19664
  • bpo-40334: Fix errors in parse_string.c with old compilers #19666
  • bpo-40334: Fix builds outside the source directory and regenerate autoconf files #19667
  • bpo-40334: Improve various PEG-Parser related stuff #19669
  • bpo-40334: Rename PyConfig.use_peg to _use_peg_parser #19670
  • bpo-40334: Compile extensions in test_peg_generator with C99 #19668
  • bpo-40334: Don't downcast from Py_ssize_t to int #19671
  • bpo-40334: Fix build errors and warnings in test_peg_generator #19672
  • bpo-40334: Suppress all output in test_peg_generator #19675
  • bpo-40334: Allow to run make regen-pegen without distutils #19684
  • bpo-40334: Use old compiler when compile mode is func_type #19692
  • bpo-40334: Rewrite test_c_parser to avoid memory leaks #19694
  • bpo-40334: Fix broken mkdir -p call in regen-pegen #19695
  • bpo-40334: Add What's New sections for PEP 617 and PEP 585 #19704
  • bpo-40334: Support CO_FUTURE_BARRY_AS_BDFL in the new parser #19721
  • bpo-40334: Add tests/test_peg_generator to the install files target #19723
  • bpo-40334: Support PyPARSE_DONT_IMPLY_DEDENT in the new parser #19736
  • bpo-40334: Catch E_EOF error when the tokenizer returns ERRORTOKEN #19743
  • bpo-40334: Don't skip test_parser:test_trigger_memory_error #19744
  • bpo-40334: Refactor peg_generator to receive a Tokens file when building c code #19745
  • bpo-40334: Fix shifting of nested fstrings #19771
  • bpo-40334: Disallow invalid single statements in the new parser #19774
  • bpo-40334: refactor and cleanup for the PEG generators #19775
  • bpo-40334: Fix test_peg_parser to actually use the old parser #19778
  • bpo-40334: Explicitly cast to int in pegen.c to fix a compiler warning #19779
  • bpo-40334: Support type comments #19780
  • bpo-40334: Improve column offsets for thrown syntax errors by Pegen #19782
  • bpo-40334: Support Python 3.6 in the PEG generator #19786
  • bpo-40334: Simplify type handling in the PEG c_generator #19818
  • bpo-40334: Add support for feature_version in new PEG parser #19827
  • bpo-40334: Ensure that tok->type_comments is set on every path #19828
  • bpo-40334: Refactor lambda_parameters similar to parameters #19830
  • bpo-40334: Correct return value of func_type_comment #19833
  • bpo-40334: unskip test_function_type in test_unparse with the new parser #19837
  • bpo-40334: Make the PyPegen* and PyParser* APIs more consistent #19839
  • bpo-40334: use the TOKENS file when checking dangling rules #19849
  • bpo-40334: regenerate metaparser as part of regen-all #19854
  • bpo-40334: Spacialized error message for invalid args after bare '*' #19865
  • bpo-40334: Set error_indicator in _PyPegen_raise_error #19887
  • bpo-40334: produce specialised errors for del #19911
  • bpo-40334: Fix error location upon parsing an invalid string literal #19962
  • bpo-40334: Add type to the assignment rule in the grammar file #19963
  • bpo-40334: Allow trailing comma in parenthesized context managers #19964
  • bpo-40334: Generate comments in the parser code to improve debugging #19966
  • bpo-40939: Add the new grammar to the grammar specification documentation #19969
  • bpo-40334: Error message for invalid default args in function call #19973
  • bpo-40334: Avoid collisions between parser variables and grammar variables #19987
  • bpo-40334: Improvements to error-handling code in the PEG parser #20003
  • bpo-40334: Do not show error caret if RAISE_SYNTAX_ERROR_NO_COL_OFFSE… #20020
  • bpo-40334: Always show the caret on SyntaxErrors #20050
  • bpo-40334: Correctly identify invalid target in assignment errors #20076
  • bpo-40334: Produce better error messages on invalid targets #20106
  • bpo-40334: Reproduce error message for type comments on bare '*' #20151
  • bpo-40334: Produce better error messages for non-parenthesized genexps #20153
  • bpo-40334: Correctly generate C parser when assigned var is None #20296
  • [3.9] bpo-40334: Produce better error messages for non-parenthesized genexps (GH-20153) #20307
  • bpo-40334: Support suppressing of multiple optional variables in Pegen #20367
  • [3.9] bpo-40334: Support suppressing of multiple optional variables in Pegen (GH-20367) #20368
  • [3.9] bpo-40334: Produce better error messages on invalid targets (GH-20106) #20973
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2020-06-19.07:54:58.363>
created_at = <Date 2020-04-19.20:16:17.317>
labels = ['type-feature', '3.9']
title = 'PEP 617: new PEG-based parser'
updated_at = <Date 2020-06-19.07:54:58.362>
user = 'https://github.com/gvanrossum'

    bugs.python.org fields:

    activity = <Date 2020-06-19.07:54:58.362>
actor = 'lys.nikolaou'
assignee = 'none'
closed = True
closed_date = <Date 2020-06-19.07:54:58.363>
closer = 'lys.nikolaou'
components = []
creation = <Date 2020-04-19.20:16:17.317>
creator = 'gvanrossum'
dependencies = []
files = []
hgrepos = []
issue_num = 40334
keywords = ['patch']
message_count = 110.0
messages = ['366803', '367009', '367050', '367057', '367059', '367060', '367061', '367063', '367064', '367066', '367067', '367068', '367069', '367071', '367073', '367075', '367076', '367084', '367097', '367105', '367112', '367113', '367114', '367115', '367121', '367126', '367135', '367136', '367137', '367139', '367140', '367141', '367142', '367155', '367158', '367160', '367170', '367171', '367190', '367191', '367196', '367237', '367243', '367246', '367439', '367440', '367448', '367467', '367470', '367471', '367473', '367474', '367521', '367564', '367576', '367577', '367580', '367582', '367598', '367602', '367610', '367637', '367713', '367772', '367815', '367822', '367835', '367836', '367839', '367840', '367843', '367845', '367850', '367854', '367862', '367866', '367891', '367914', '367916', '367974', '367976', '367978', '368004', '368034', '368288', '368302', '368304', '368329', '368330', '368566', '368567', '368593', '368595', '368598', '368599', '368663', '368796', '368892', '369090', '369166', '369203', '369288', '369536', '369553', '369554', '369837', '369840', '371841', '371842', '371848']
nosy_count = 11.0
nosy_names = ['gvanrossum', 'rhettinger', 'ned.deily', 'serhiy.storchaka', 'mbussonn', 'Anthony Sottile', 'lys.nikolaou', 'pablogsal', 'miss-islington', 'BTaskaya', 'hauntsaninja']
pr_nums = ['19503', '19664', '19666', '19667', '19669', '19670', '19668', '19671', '19672', '19675', '19684', '19692', '19694', '19695', '19704', '19721', '19723', '19736', '19743', '19744', '19745', '19771', '19774', '19775', '19778', '19779', '19780', '19782', '19786', '19818', '19827', '19828', '19830', '19833', '19837', '19839', '19849', '19854', '19865', '19887', '19911', '19962', '19963', '19964', '19966', '19969', '19973', '19987', '20003', '20020', '20050', '20076', '20106', '20151', '20153', '20296', '20307', '20367', '20368', '20973']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue40334'
versions = ['Python 3.9']

    Activity

    1. gvanrossum commented on Apr 19, 2020

      @gvanrossum
      gvanrossum
      on Apr 19, 2020
      MemberAuthor

      Note: PEP-617 is currently under review by the Steering Council, but if they approve we'd like to get it into alpha 6, and reviews are welcome (even though we're still finagling some corner cases of the implementation).

    2. gvanrossum
      added
      3.9 (EOL)end of life
      on Apr 19, 2020

    3. vstinner commented on Apr 22, 2020

      @vstinner
      vstinner
      on Apr 22, 2020
      Member

      Once PR 19503 will be merged, I would be interested to see if setjmp()/longjmp() can be avoided. I'm scared by these functions: they require that all functions in between setjmp() and longjmp() call stack have no state and don't need any cleanup at exit.

    4. gvanrossum commented on Apr 22, 2020

      @gvanrossum
      gvanrossum
      on Apr 22, 2020
      MemberAuthor

      VIctor, you were very right about longjmp. See we-like-parsers#119

    5. pablogsal commented on Apr 22, 2020

      @pablogsal
      pablogsal
      on Apr 22, 2020
      Member

      New changeset c5fc156 by Pablo Galindo in branch 'master':
      bpo-40334: PEP-617 implementation: New PEG parser for CPython (GH-19503)
      c5fc156

    6. pablogsal commented on Apr 22, 2020

      @pablogsal
      pablogsal
      on Apr 22, 2020
      Member

      New changeset 458004b by Pablo Galindo in branch 'master':
      bpo-40334: Fix errors in parse_string.c with old compilers (GH-19666)
      458004b

    7. vstinner commented on Apr 22, 2020

      @vstinner
      vstinner
      on Apr 22, 2020
      Member

      bpo-40334: Fix errors in parse_string.c with old compilers (GH-19666)

      Pablo told me in private that he plans to add a buildbot worker to test the old parser. So this change is fine.

    8. ned-deily commented on Apr 22, 2020

      @ned-deily
      ned-deily
      on Apr 22, 2020
      Member

      Looks like the build changes do not work with a build directory outside of the source directory:

      mkdir build && cd build && ../configure && make

      gcc -c -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -I../Include/internal -IObjects -IInclude -IPython -I. -I../Include -DPy_BUILD_CORE -o Parser/pegen/pegen.o ../Parser/pegen/pegen.c
      error: unable to open output file 'Parser/pegen/pegen.o': 'No such file or directory'
      1 error generated.
      make: *** [Parser/pegen/pegen.o] Error 1

      Lots of build processes depend on this, like for putting out a release.

    9. pablogsal commented on Apr 22, 2020

      @pablogsal
      pablogsal
      on Apr 22, 2020
      Member

      Looks like the build changes do not work with a build directory outside of the source directory:

      Opened #19667. Ned, could you review?

    10. pablogsal commented on Apr 23, 2020

      @pablogsal
      pablogsal
      on Apr 23, 2020
      Member

      New changeset a25f3c4 by Pablo Galindo in branch 'master':
      bpo-40334: Fix builds outside the source directory and regenerate autoconf files (GH-19667)
      a25f3c4

    11. vstinner commented on Apr 23, 2020

      @vstinner
      vstinner
      on Apr 23, 2020
      Member

      New changeset 1def775 by Victor Stinner in branch 'master':
      bpo-40334: Rename PyConfig.use_peg to _use_peg_parser (GH-19670)
      1def775

    12. vstinner commented on Apr 23, 2020

      @vstinner
      vstinner
      on Apr 23, 2020
      Member

      bpo-40334: Rename PyConfig.use_peg to _use_peg_parser (GH-19670)

      This change removes sys.flags.use_peg which was only used in tests. If someone sees a good reason, we may add it back. In the meanwhile, test.support.use_old_parser() can be used.

      I didn't like the idea of having a new sys.flags.use_peg flag appears in 3.9 but disappears in 3.10.

      FYI sys.flags still supports the tuple API! Example:

      >>> sys.flags[9]
0

      I have no idea what is this 10th member of sys.flags :-)

    13. vstinner commented on Apr 23, 2020

      @vstinner
      vstinner
      on Apr 23, 2020
      Member

      test_peg_generator emits compiler warnings:

      https://buildbot.python.org/all/#/builders/612/builds/283

      0:08:01 load avg: 1.40 [423/423/1] test_peg_generator passed (7 min 4 sec)
      /tmp/tmp0vbqe8gp/parse.c: In function ‘start_rule’:
      /tmp/tmp0vbqe8gp/parse.c:32:35: warning: passing argument 2 of ‘_PyPegen_lookahead’ from incompatible pointer type [-Wincompatible-pointer-types]
      32 | _PyPegen_lookahead(1, _PyPegen_name_token, p)
      | ^~~~~~~~~~~~~~~~~~~
      | |
      | struct _expr * ()(Parser *)
      In file included from /tmp/tmp0vbqe8gp/parse.c:2:
      /home/buildbot/buildarea/3.x.cstratak-fedora-rawhide-aarch64.lto-pgo/build/Parser/pegen/pegen.h:90:36: note: expected ‘void * (      )(Parser *)’ but argument is of type ‘struct _expr * ()(Parser *)’
      90 | int _PyPegen_lookahead(int, void *(func)(Parser *), Parser *);
      | ~~~~~~~^~~~~~~~~~~~~~~
      /tmp/tmp54o68k3n/parse.c: In function ‘start_rule’:
      /tmp/tmp54o68k3n/parse.c:32:35: warning: passing argument 2 of ‘_PyPegen_lookahead’ from incompatible pointer type [-Wincompatible-pointer-types]
      32 | _PyPegen_lookahead(0, _PyPegen_name_token, p)
      | ^~~~~~~~~~~~~~~~~~~
      | |
      | struct _expr * (      )(Parser *)
      In file included from /tmp/tmp54o68k3n/parse.c:2:
      /home/buildbot/buildarea/3.x.cstratak-fedora-rawhide-aarch64.lto-pgo/build/Parser/pegen/pegen.h:90:36: note: expected ‘void * ()(Parser *)’ but argument is of type ‘struct _expr * ()(Parser *)’
      90 | int _PyPegen_lookahead(int, void *(func)(Parser *), Parser *);
      | ~~~~~~~^~~~~~~~~~~~~~~

    14. lysnikolaou commented on Apr 23, 2020

      @lysnikolaou
      lysnikolaou
      on Apr 23, 2020
      Member

      test_peg_generator emits compiler warnings

      These are expected. Is this a problem?

    15. 91 remaining items

    16. pablogsal commented on May 17, 2020

      @pablogsal
      pablogsal
      on May 17, 2020
      Member

      New changeset 2c8cd06 by Lysandros Nikolaou in branch 'master':
      bpo-40334: Improvements to error-handling code in the PEG parser (GH-20003)
      2c8cd06

    17. rhettinger commented on May 18, 2020

      @rhettinger
      rhettinger
      on May 18, 2020
      Contributor

      One minor nit: We're getting a nuisance compiler warning:

      test_peg_generator passed (2 min 40 sec)
      /var/folders/1b/3499qp8s7xv5w0fvjcgl45100000gn/T/tmp90rifd5z/parse.c:97:19: warning: code will never be executed [-Wunreachable-code]
      p->mark = _mark;
      ^~~~~
      1 warning generated.

    18. lysnikolaou commented on May 18, 2020

      @lysnikolaou
      lysnikolaou
      on May 18, 2020
      Member

      I'm not getting any compiler warnings on macOS (clang) and on Ubuntu (gcc) and I can't find any related warnings on the Windows Github Actions logs either. Could you maybe post a verbose log of test_peg_generator, Raymond?

    19. pablogsal commented on May 18, 2020

      @pablogsal
      pablogsal
      on May 18, 2020
      Member

      New changeset 75b863a by Lysandros Nikolaou in branch 'master':
      bpo-40334: Reproduce error message for type comments on bare '*' in the new parser (GH-20151)
      75b863a

    20. pablogsal commented on May 21, 2020

      @pablogsal
      pablogsal
      on May 21, 2020
      Member

      New changeset f50516e by Batuhan Taskaya in branch 'master':
      bpo-40334: Correctly generate C parser when assigned var is None (GH-20296)
      f50516e

    21. pablogsal commented on May 22, 2020

      @pablogsal
      pablogsal
      on May 22, 2020
      Member

      New changeset ae14583 by Lysandros Nikolaou in branch 'master':
      bpo-40334: Produce better error messages for non-parenthesized genexps (GH-20153)
      ae14583

    22. miss-islington commented on May 22, 2020

      @miss-islington
      miss-islington
      on May 22, 2020
      Contributor

      New changeset 55c8923 by Miss Islington (bot) in branch '3.9':
      bpo-40334: Produce better error messages for non-parenthesized genexps (GH-20153)
      55c8923

    23. pablogsal commented on May 24, 2020

      @pablogsal
      pablogsal
      on May 24, 2020
      Member

      New changeset cba5031 by Batuhan Taskaya in branch 'master':
      bpo-40334: Support suppressing of multiple optional variables in Pegen (GH-20367)
      cba5031

    24. miss-islington commented on May 24, 2020

      @miss-islington
      miss-islington
      on May 24, 2020
      Contributor

      New changeset 82c274e by Miss Islington (bot) in branch '3.9':
      bpo-40334: Support suppressing of multiple optional variables in Pegen (GH-20367)
      82c274e

    25. pablogsal commented on Jun 18, 2020

      @pablogsal
      pablogsal
      on Jun 18, 2020
      Member

      New changeset 01ece63 by Lysandros Nikolaou in branch 'master':
      bpo-40334: Produce better error messages on invalid targets (GH-20106)
      01ece63

    26. pablogsal commented on Jun 19, 2020

      @pablogsal
      pablogsal
      on Jun 19, 2020
      Member

      New changeset a5442b2 by Lysandros Nikolaou in branch '3.9':
      [3.9] bpo-40334: Produce better error messages on invalid targets (GH-20106) (GH-20973)
      a5442b2

    27. lysnikolaou commented on Jun 19, 2020

      @lysnikolaou
      lysnikolaou
      on Jun 19, 2020
      Member

      All parts of the implementation are now in. New issues should be opened for any potential bugs.

    28. lysnikolaou
      closed this as completedon Jun 19, 2020
    29. lysnikolaou
      closed this as completedon Jun 19, 2020
    30. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.9 (EOL)end of lifetype-featureA feature request or enhancement

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions