Skip to content

[WIP] PEP 511: Add sys.set_code_transformers() #70333

Closed
Closed
[WIP] PEP 511: Add sys.set_code_transformers()#70333
Labels
type-featureA feature request or enhancement

Description

@vstinner
vstinner
opened on Jan 18, 2016
BPO 26145
Nosy @brettcannon, @vstinner, @1st1
PRs
  • [WIP] bpo-26145: Implement PEP 511, code transformer #2355
    		•
    Files
  • transformers.patch
  • test_sys.patch
  • transformers-2.patch
  • transformers-3.patch
  • transformers-4.patch
  • pycf_transformed_ast.patch
  • transformers-5.patch
  • transformers-6.patch
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2017-10-17.20:25:30.072>
created_at = <Date 2016-01-18.09:10:57.897>
labels = ['type-feature']
title = '[WIP] PEP 511: Add sys.set_code_transformers()'
updated_at = <Date 2019-06-14.14:13:43.963>
user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2019-06-14.14:13:43.963>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2017-10-17.20:25:30.072>
closer = 'vstinner'
components = []
creation = <Date 2016-01-18.09:10:57.897>
creator = 'vstinner'
dependencies = []
files = ['41643', '41644', '41656', '41692', '41695', '41700', '41816', '41912']
hgrepos = []
issue_num = 26145
keywords = ['patch']
message_count = 15.0
messages = ['258509', '258511', '258560', '258573', '258580', '258581', '258678', '258822', '258829', '258866', '259648', '260217', '260222', '296704', '304529']
nosy_count = 3.0
nosy_names = ['brett.cannon', 'vstinner', 'yselivanov']
pr_nums = ['2355']
priority = 'normal'
resolution = 'rejected'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue26145'
versions = ['Python 3.6']

    Activity

    1. vstinner commented on Jan 18, 2016

      @vstinner
      vstinner
      on Jan 18, 2016
      MemberAuthor

      Attached patch implements the largest part of the PEP-511:

      • add sys.get_code_transformers() and sys.set_code_transformers()
      • add sys.implementation.optim_tag
      • add -o command line option setting sys.implementation.optim_tag
      • modify importlib to use sys.implementation.optim_tag to build .pyc filenames
      • modify importlib to raise an ImportError when the .pyc file is missing, the .py is available, and the optimizer tag created from code transformers is different than the current sys.implementation.optim_tag
      • add Lib/test/test_pep511.py: test changes especially bytecode and AST transformers
      • add a PeepholeOptimizer object created by default for sys.set_code_transformers()
      • skip the peephole optimizer when -o noopt is used
      • add sys field to the PyInterpreterState structure: used by _PySys_GetCodeTransformers() to get code transformers
      • update unit tests to use the optimizer to create the .pyc filename

      An optimizer tag must contain characters: '.', '-', directory separators or NUL character.

      Brett Canon proposed to not include the optimizer tag in .pyc filenames if it is "opt" to limit backward incompatible changes.

      Note: The PEP-511 is under discussion on python-ideas. The patch implements the first version of the PEP, sent to python-ideas, so it doesn't include proposed changes like changing code_transformer() prototype to "def code_transformer(self, code, context)".

      See also the issue bpo-26100: add test.support.optim_args_from_interpreter_flags().

    2. vstinner
      added
      type-featureA feature request or enhancement
      on Jan 18, 2016

    3. vstinner commented on Jan 18, 2016

      @vstinner
      vstinner
      on Jan 18, 2016
      MemberAuthor

      Missing patch for test_sys testing the new sys.get/set_code_transformers() functions.

    4. vstinner commented on Jan 18, 2016

      @vstinner
      vstinner
      on Jan 18, 2016
      MemberAuthor

      main.c:

      + case 'o':
      + if (wcslen(_PyOS_optarg) == 0
      + || wcschr(_PyOS_optarg, L'.')
      + || wcschr(_PyOS_optarg, L'-')
      ...

      '-' character must be valid for -o argument.

    5. vstinner commented on Jan 19, 2016

      @vstinner
      vstinner
      on Jan 19, 2016
      MemberAuthor

      sys_init_code_transformer: Py_DECREF(name); must be Py_XDECREF(name); at exit, it can NULL. An unit test on an invalid transformer (with no name) is missing?

    6. vstinner commented on Jan 19, 2016

      @vstinner
      vstinner
      on Jan 19, 2016
      MemberAuthor

      Patch version 2:

      • allow "-" character in the argument of the -o command line option
      • add an unit test on invalid type for code transformer
      • fix a crash in sys.set_code_transformers(): replace Py_DECREF() with Py_XDECREF()

    7. vstinner commented on Jan 19, 2016

      @vstinner
      vstinner
      on Jan 19, 2016
      MemberAuthor

      (oops, i posted the same message with patch twice)

    8. vstinner commented on Jan 20, 2016

      @vstinner
      vstinner
      on Jan 20, 2016
      MemberAuthor

      Note: ast.PyCF_TRANSFORMED_AST is not implemented yet.

    9. vstinner commented on Jan 22, 2016

      @vstinner
      vstinner
      on Jan 22, 2016
      MemberAuthor

      Rebased patch.

    10. vstinner commented on Jan 22, 2016

      @vstinner
      vstinner
      on Jan 22, 2016
      MemberAuthor

      Patch version 4: Fix sys_set_code_transformers(), initialize seq to NULL to fix a crash on error handling.

    11. vstinner commented on Jan 23, 2016

      @vstinner
      vstinner
      on Jan 23, 2016
      MemberAuthor

      Note: ast.PyCF_TRANSFORMED_AST is not implemented yet.

      Additionnal pycf_transformed_ast.patch implements it. The patch should be applied on top of transformers-4.patch.

      Note: PyCF_TRANSFORMED_AST has the same value (0x1000) than the old constant CO_GENERATOR_ALLOWED. This constant was removed in Python 2.5.0 by the changeset 6b42920accc9 in 2006. Is it an issue?

    12. vstinner commented on Feb 5, 2016

      @vstinner
      vstinner
      on Feb 5, 2016
      MemberAuthor

      Rebased patch combining also pycf_transformed_ast.patch.

    13. vstinner commented on Feb 13, 2016

      @vstinner
      vstinner
      on Feb 13, 2016
      MemberAuthor

      Patch version 6: updated to the new code_transformer() API, it now takes a code object as input and must return a new code object.

      Sadly, it looks like there are reference leaks (try: ./python -m test -R 3:3 test_import).

    14. vstinner commented on Feb 13, 2016

      @vstinner
      vstinner
      on Feb 13, 2016
      MemberAuthor

      TODO: implement a fast-path for PyCode_Optimize() avoiding completly the "expensive" code transformer API, before the first call to sys.set_code_transformer().

      It would avoid list <=> tuple conversions for code constants, and the need of creating two code objects.

      In practice, this fast-path should be taken in most cases.

    15. vstinner commented on Jun 23, 2017

      @vstinner
      vstinner
      on Jun 23, 2017
      MemberAuthor

      Recently, some people asked me for an update for my FAT Python project. So I rebased this change I wrote 1 year 1/2 and adapted it for the new code base:

      • I renamed test_pep511.py to test_code_transformer.py
      • I removed the sys module from the PyInterpreterState structure (added in my previous patch), since Eric Snow and Nick Coghlan removed a variant of the sys module from this structure for their work on subinterpreters and reworked Python initialization

      The PEP-511 is not accepted, so the implementation is still a work-in-progress (WIP) and must not be merged.

    16. vstinner
      changed the title [-]PEP 511: Add sys.set_code_transformers()[/-] [+][WIP] PEP 511: Add sys.set_code_transformers()[/+] on Jun 28, 2017

    17. vstinner commented on Oct 17, 2017

      @vstinner
      vstinner
      on Oct 17, 2017
      MemberAuthor

      I rejected my own PEP-511, so I now reject this issue as well.
      https://mail.python.org/pipermail/python-dev/2017-October/149903.html

    18. vstinner
      closed this as completedon Oct 17, 2017
    19. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      type-featureA feature request or enhancement

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions