Skip to content

Remove the old parser #85111

Closed
Closed
Remove the old parser#85111
Assignees
pablogsal
Labels
3.10 (EOL)end of lifeinterpreter-core(Objects, Python, Grammar, and Parser dirs)

Description

@pablogsal
pablogsal
opened on Jun 10, 2020
BPO 40939
Nosy @gvanrossum, @terryjreedy, @vstinner, @ned-deily, @encukou, @nnemkin, @hroncok, @lysnikolaou, @pablogsal, @miss-islington, @brandtbucher
PRs
  • bpo-40939: Remove the old parser #20768
  • bpo-40939: Generate keyword.py using pegen #20800
  • [3.9] bpo-40939: Generate keyword.py using the new parser (GH-20800) #20801
  • bpo-40939: Remove PEG parser easter egg #20802
  • [3.9] bpo-40939: Fix test_keyword for the old parser #20814
  • bpo-40939: Remove some extra references to PYTHONOLDPARSER #20815
  • bpo-40939: Clean and adapt the peg_generator directory after deleting the old parser #20822
  • bpo-40939: Remove the old parser (Part 2) #21005
  • [3.9] bpo-40939: Deprecate the PyParser_SimpleParse* functions #21012
  • bpo-40939: Rename PyPegen* functions to PyParser* #21016
  • [3.9] bpo-40939: Do not emit deprecation warnings inside CPython for old parser APIs #21025
  • [3.9] bpo-40939: Deprecate PyNode_Compile #21036
  • bpo-40939: run autoreconf to fix configure{.ac,} disparity #21152
  • bpo-40939: Delete remaining references to Grammar/Grammar #21624
  • bpo-40939: Add the new grammar to the grammar specification documentation #19969
  • [3.9] bpo-40939: Use the new grammar for the grammar specification documentation (GH-19969) #21641
  • bpo-40939: Remove even more references to the old parser #21642
  • bpo-41808: Add What's New 3.9 entry missing from master #22294
  • bpo-40939: Document removal of the old parser in 3.10 whatsnew #23321
  • bpo-40939: Restore some stable API functions incorrectly deleted #23606
  • bpo-40939: Remove documentation for PyParser_* & add porting notes #26855
  • [3.10] bpo-40939: Remove documentation for PyParser_* & add porting notes (GH-26855) #26898
    		•
    Files
  • py_limited_api_example.zip: Example of using Py_CompileString with Py_LIMITED_API
  • py3.txt: Imports from python3.dll
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/pablogsal'
closed_at = <Date 2020-12-02.05:17:21.232>
created_at = <Date 2020-06-10.11:25:17.640>
labels = ['interpreter-core', '3.10']
title = 'Remove the old parser'
updated_at = <Date 2021-06-28.09:36:20.843>
user = 'https://github.com/pablogsal'

    bugs.python.org fields:

    activity = <Date 2021-06-28.09:36:20.843>
actor = 'petr.viktorin'
assignee = 'pablogsal'
closed = True
closed_date = <Date 2020-12-02.05:17:21.232>
closer = 'pablogsal'
components = ['Interpreter Core']
creation = <Date 2020-06-10.11:25:17.640>
creator = 'pablogsal'
dependencies = []
files = ['49642', '49647']
hgrepos = []
issue_num = 40939
keywords = ['patch']
message_count = 69.0
messages = ['371182', '371264', '371265', '371266', '371267', '371268', '371270', '371271', '371288', '371289', '371290', '371291', '371292', '371301', '371307', '371308', '371310', '371311', '371316', '371327', '371929', '371940', '371942', '371953', '371977', '372023', '372025', '372026', '372027', '372370', '374282', '374346', '374410', '374413', '374416', '374417', '377083', '380826', '380828', '380855', '380863', '381095', '381103', '381112', '381136', '381137', '381140', '381172', '381176', '381182', '381227', '381232', '382129', '382130', '382134', '382285', '382287', '382289', '382293', '382299', '382323', '382324', '382325', '382329', '382337', '387017', '396348', '396488', '396622']
nosy_count = 12.0
nosy_names = ['gvanrossum', 'terry.reedy', 'vstinner', 'ned.deily', 'petr.viktorin', 'nnemkin', 'Igor.Skochinsky', 'hroncok', 'lys.nikolaou', 'pablogsal', 'miss-islington', 'brandtbucher']
pr_nums = ['20768', '20800', '20801', '20802', '20814', '20815', '20822', '21005', '21012', '21016', '21025', '21036', '21152', '21624', '19969', '21641', '21642', '22294', '23321', '23606', '26855', '26898']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue40939'
versions = ['Python 3.10']

    Activity

    1. pablogsal commented on Jun 10, 2020

      @pablogsal
      pablogsal
      on Jun 10, 2020
      MemberAuthor

      As stated in PEP-617, the old parser will be removed in Python 3.10

    2. pablogsal
      added
      3.10 (EOL)end of life
      on Jun 10, 2020
    3. pablogsal
      self-assigned this
      on Jun 10, 2020
    4. pablogsal
      added
      interpreter-core(Objects, Python, Grammar, and Parser dirs)
      3.10 (EOL)end of life
      on Jun 10, 2020
    5. pablogsal
      self-assigned this
      on Jun 10, 2020
    6. pablogsal
      added
      interpreter-core(Objects, Python, Grammar, and Parser dirs)
      on Jun 10, 2020

    7. pablogsal commented on Jun 11, 2020

      @pablogsal
      pablogsal
      on Jun 11, 2020
      MemberAuthor

      New changeset 9727694 by Lysandros Nikolaou in branch 'master':
      bpo-40939: Generate keyword.py using the new parser (GH-20800)
      9727694

    8. vstinner commented on Jun 11, 2020

      @vstinner
      vstinner
      on Jun 11, 2020
      Member
      >>> __new_parser__
  File "<stdin>", line 1
    __new_parser__
    ^
SyntaxError: You found it!

      "new", "ex" or "ng" are not really future proof names. Can we rename the keyword to "__peg_parser__"?

    9. lysnikolaou commented on Jun 11, 2020

      @lysnikolaou
      lysnikolaou
      on Jun 11, 2020
      Member

      Can we rename the keyword to "__peg_parser__"?

      I guess we could just remove this, as soon as the old parser is out. We were only using this to differentiate between the two parsers, when we were testing enabling/disabling the old one. I could get a PR ready to be merged after #64967 is there.

    10. pablogsal commented on Jun 11, 2020

      @pablogsal
      pablogsal
      on Jun 11, 2020
      MemberAuthor

      I guess we could just remove this, as soon as the old parser is out. We were only using this to differentiate between the two parsers, when we were testing enabling/disabling the old one. I could get a PR ready to be merged after #64967 is there.

      I would personally would like to keep the easter egg, but I assume is better to rename it to "__peg_parser__".

    11. miss-islington commented on Jun 11, 2020

      @miss-islington
      miss-islington
      on Jun 11, 2020
      Contributor

      New changeset 961edf7 by Miss Islington (bot) in branch '3.9':
      bpo-40939: Generate keyword.py using the new parser (GH-20800)
      961edf7

    12. lysnikolaou commented on Jun 11, 2020

      @lysnikolaou
      lysnikolaou
      on Jun 11, 2020
      Member

      I would personally would like to keep the easter egg, but I assume is better to rename it to "__peg_parser__".

      Ok then! On it.

    13. vstinner commented on Jun 11, 2020

      @vstinner
      vstinner
      on Jun 11, 2020
      Member

      I did'nt ask to remove the easter egg. I'm just asking to avoid the "new" name. In my experience, each time that a "new" thing happens, later we have to use "new extended", "new_v2" or worse name :-)

      Oh, if the name changes, please change it in 3.9 as well.

      Look at this amazing names of the 5 flavors of functions parsing a string:

      PyParser_ParseString()
PyParser_ParseStringFlags()
PyParser_ParseStringFlagsFilename()
PyParser_ParseStringFlagsFilenameEx() <= public!
PyParser_ParseStringObject()

      Same for parsing a file:

      PyParser_ParseFile()
PyParser_ParseFileFlags()
PyParser_ParseFileFlagsEx() <= public!
PyParser_ParseFileObject()

      Or PyRun functions:

      PyRun_String()
PyRun_AnyFile()
PyRun_AnyFileEx() <= public!
PyRun_AnyFileFlags()
PyRun_SimpleString()
PyRun_SimpleFile()
PyRun_SimpleFileEx() <= public!
PyRun_InteractiveOne()
PyRun_InteractiveLoop()
PyRun_File()
PyRun_FileEx() <= public!
PyRun_FileFlags()

      ceval.c:

      PyEval_EvalCode()
PyEval_EvalCodeEx() <= public!
_PyEval_EvalCodeWithName()
_PyEval_EvalCode()

      I cannot count the number of "Ex" functions that we have :-)

      Py_Finalize() -> :Py_FinalizeEx() <= public!
      PyErr_Print() -> PyErr_PrintEx() <= public!
      PySys_SetArgv() -> PySys_SetArgvEx() <= public!
      PyErr_WarnEx() <= public!

      _PyBytes_FormatEx()
_PyDict_MergeEx()
_Py_DecodeLocaleEx(), _Py_EncodeLocaleEx()
struct PyMemAllocatorEx
_Py_DecodeUTF8Ex(), _Py_EncodeUTF8Ex()
etc.

    14. gvanrossum commented on Jun 11, 2020

      @gvanrossum
      gvanrossum
      on Jun 11, 2020
      Member

      Honestly I see no reason to keep that easter egg. Can we remove it please?

    15. 52 remaining items

    16. pablogsal commented on Dec 2, 2020

      @pablogsal
      pablogsal
      on Dec 2, 2020
      MemberAuthor

      Seems that commit 1ed83ad was deleting some functions that were not correctly covered by redirection macros. I have opened PR 23606 to restore those as they were.

    17. pablogsal commented on Dec 2, 2020

      @pablogsal
      pablogsal
      on Dec 2, 2020
      MemberAuthor

      New changeset 46bd5ed by Pablo Galindo in branch 'master':
      bpo-40939: Restore some stable API functions incorrectly deleted (GH-23606)
      46bd5ed

    18. pablogsal
      closed this as completedon Dec 2, 2020
    19. pablogsal
      closed this as completedon Dec 2, 2020

    20. IgorSkochinsky commented on Dec 2, 2020

      @IgorSkochinsky
      IgorSkochinskymannequin
      on Dec 2, 2020
      Mannequin

      Thanks for the fix! I'd like to submit a test to avoid this and similar issues in future. Are there any guidelines for this? Sorry if this is a wrong place to ask.

    21. pablogsal commented on Dec 2, 2020

      @pablogsal
      pablogsal
      on Dec 2, 2020
      MemberAuthor

      I am currently working on test that checks that the stable API symbols are correctly exported. Unfortunately there is no official maintained list if those symbols, so is taking a while

    22. IgorSkochinsky commented on Dec 2, 2020

      @IgorSkochinsky
      IgorSkochinskymannequin
      on Dec 2, 2020
      Mannequin

      I am currently working on test that checks that the stable API symbols are correctly exported.

      Thank you very much! For added motivation, the 3.8.0 release was unusable thanks to bpo-37633 which was somewhat similar (also in pythonrun). For reference, I've attached a list of symbols we currently use (a few more we have to import dynamically since they're not in the stable ABI but we'd like to keep that list as short as possible).

    23. nnemkin commented on Dec 2, 2020

      @nnemkin
      nnemkinmannequin
      on Dec 2, 2020
      Mannequin

      Unfortunately there is no official maintained list if those symbols, so is taking a while

      On Windows there is PC/python3dll.c.

    24. pablogsal commented on Dec 2, 2020

      @pablogsal
      pablogsal
      on Dec 2, 2020
      MemberAuthor

      On Windows there is PC/python3dll.c.

      Even that is severely out of date, unfortunately. There are many functions that were removed in 3.8 and 3.7 that are still listed there.

    25. pablogsal commented on Dec 2, 2020

      @pablogsal
      pablogsal
      on Dec 2, 2020
      MemberAuthor

      Opened https://bugs.python.org/issue42545 for that

    26. vstinner commented on Feb 15, 2021

      @vstinner
      vstinner
      on Feb 15, 2021
      Member

      FYI the unbound project was fixed by calling Py_CompileString() on Python 3.9 and newer:
      NLnetLabs/unbound@e0d426e

    27. encukou commented on Jun 22, 2021

      @encukou
      encukou
      on Jun 22, 2021
      Member

      • The removed functions are still listed in the documentation; this is confusing.

      • struct _node is not very usable in the C API, but what I saw in mod_wsgi was giving gave PyParser_* output to PyNode_Compile. I tried to write porting notes for this usage. It should also help with the "compile to check for syntax errors" case in unbound.

    28. encukou commented on Jun 24, 2021

      @encukou
      encukou
      on Jun 24, 2021
      Member

      New changeset 29987f7 by Petr Viktorin in branch 'main':
      bpo-40939: Remove documentation for PyParser_* & add porting notes (GH-26855)
      29987f7

    29. encukou commented on Jun 28, 2021

      @encukou
      encukou
      on Jun 28, 2021
      Member

      New changeset dc10264 by Miss Islington (bot) in branch '3.10':
      bpo-40939: Remove documentation for PyParser_* & add porting notes (GH-26855) (GH-26898)
      dc10264

    30. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    3.10 (EOL)end of lifeinterpreter-core(Objects, Python, Grammar, and Parser dirs)

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions