Skip to content

Remove the usage of the C stack in Python to Python calls #89419

Closed
Closed
Remove the usage of the C stack in Python to Python calls#89419
Labels
3.11only security fixes

Description

@pablogsal
pablogsal
opened on Sep 21, 2021
BPO 45256
Nosy @gvanrossum, @pmp-p, @markshannon, @serhiy-storchaka, @zooba, @ctismer, @corona10, @pablogsal, @brandtbucher, @Fidget-Spinner, @thesamesam
PRs
  • bpo-45256: Remove the usage of the C stack in Python to Python calls #28488
  • bpo-45256: Small cleanups for the code that inlines Python-to-Python calls in ceval.c #28836
  • bpo-45256: Fix cleanup of stolen locals for Python-to-Python calls #28905
  • bpo-45256: Avoid C calls for more Python to Python calls. #28937
  • bpo-45256: Rationalize code around Python-to-Python calls a bit. #29235
  • bpo-45256: Don't call at the C level when calling bound-methods from Python code. #29238
  • bpo-45256: Don't track the exact depth of each InterpreterFrame #30372
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2022-01-06.12:52:24.256>
created_at = <Date 2021-09-21.10:41:59.834>
labels = ['3.11']
title = 'Remove the usage of the C stack in Python to Python calls'
updated_at = <Date 2022-01-06.12:52:24.256>
user = 'https://github.com/pablogsal'

    bugs.python.org fields:

    activity = <Date 2022-01-06.12:52:24.256>
actor = 'Mark.Shannon'
assignee = 'none'
closed = True
closed_date = <Date 2022-01-06.12:52:24.256>
closer = 'Mark.Shannon'
components = []
creation = <Date 2021-09-21.10:41:59.834>
creator = 'pablogsal'
dependencies = []
files = []
hgrepos = []
issue_num = 45256
keywords = ['patch']
message_count = 21.0
messages = ['402311', '402429', '402430', '402485', '402491', '402504', '402518', '402546', '402763', '402766', '402767', '402780', '402785', '403538', '403543', '403832', '404166', '405109', '405193', '406462', '409753']
nosy_count = 11.0
nosy_names = ['gvanrossum', 'pmpp', 'Mark.Shannon', 'serhiy.storchaka', 'steve.dower', 'Christian.Tismer', 'corona10', 'pablogsal', 'brandtbucher', 'kj', 'thesamesam']
pr_nums = ['28488', '28836', '28905', '28937', '29235', '29238', '30372']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue45256'
versions = ['Python 3.11']

    Linked PRs

    Activity

    1. pablogsal commented on Sep 21, 2021

      @pablogsal
      pablogsal
      on Sep 21, 2021
      MemberAuthor

      Removing the usage of the C stack in Python-to-Python calls will allow future optimizations in the eval loop to take place and can yield some speed ups given that we will be removing C function calls and preambles by inlining the callee in the same eval loop as the caller.

    2. pablogsal
      added
      3.11only security fixes
      on Sep 21, 2021

    3. serhiy-storchaka commented on Sep 22, 2021

      @serhiy-storchaka
      serhiy-storchaka
      on Sep 22, 2021
      Member

      AFAIK Mark Shannon proposed this idea, but it was rejected.

    4. pablogsal commented on Sep 22, 2021

      @pablogsal
      pablogsal
      on Sep 22, 2021
      MemberAuthor

      What was rejected was https://www.python.org/dev/peps/pep-0651/ which included this idea but had a lot more stuff in it. In particular, it was rejected because it gave semantics to overflow exceptions (two exceptions were proposed), new APIs and it had lack consistent guarantees for different platforms, among other considerations.

      This version is just for optimization purposes with no changes in semantics.

    5. vstinner
      changed the title [-]Remove the usage of the cstack in Python to Python calls[/-] [+]Remove the usage of the C stack in Python to Python calls[/+] on Sep 22, 2021
    6. vstinner
      changed the title [-]Remove the usage of the cstack in Python to Python calls[/-] [+]Remove the usage of the C stack in Python to Python calls[/+] on Sep 22, 2021

    7. ctismer commented on Sep 23, 2021

      @ctismer
      ctismer
      on Sep 23, 2021
      Contributor

      Hey guys, you know that you are about to implement the core idea of Stackless Python, right? :-D

    8. markshannon commented on Sep 23, 2021

      @markshannon
      markshannon
      on Sep 23, 2021
      Member

      I've trying to do this since about 2011 :)

    9. zooba commented on Sep 23, 2021

      @zooba
      zooba
      on Sep 23, 2021
      Member

      I fully support implementing the core idea of Stackless Python :)

      I spent a whole EuroPython a couple of years back discussing the idea with (apparently) everyone except Mark.

      Though I wouldn't like to lose the ability to extract the Python stack by inspecting native memory alone. That is very useful for debugging, particularly when you've only got a crash dump. So provided all the code objects are only an indirection or two away from a local (or better yet, parameter) value, it should be fine.

    10. pablogsal commented on Sep 23, 2021

      @pablogsal
      pablogsal
      on Sep 23, 2021
      MemberAuthor

      Though I wouldn't like to lose the ability to extract the Python stack by inspecting native memory alone.

      Don't worry about it, I am personally making sure that keeps being possible. it will need some changes in the tools, but not any more that any other changes between minor versions of the interpreter

    11. ctismer commented on Sep 24, 2021

      @ctismer
      ctismer
      on Sep 24, 2021
      Contributor

      FYI., in Stackless Python I built a chain of frames by a double linked list. This was the replacement for the current frame/stack mix.
      Debugging was still quite easy, following this frame chain.

      Note that it is a rather easy step to add the capability to squirrel the whole current chain away and replace it by another one. Add some header to such a chain, and you can call it "Tasklet".

    12. markshannon commented on Sep 28, 2021

      @markshannon
      markshannon
      on Sep 28, 2021
      Member

      PR 28488 has no NEWS entry, or What's New entry.

      However, adding multiple entries will be confusing, so that's best left until all calls to Python functions and method don't use the C stack.

    13. ctismer commented on Sep 28, 2021

      @ctismer
      ctismer
      on Sep 28, 2021
      Contributor

      FWIW, getting all function to avoid the C stack will most probably take a long time, if it happens at all. Especially functions which might call into Python multiple times must be re-designed heavily, turned into multiple pieces to be in tail position. Been there, long ago... :)

    14. pablogsal commented on Sep 28, 2021

      @pablogsal
      pablogsal
      on Sep 28, 2021
      MemberAuthor

      We are starting to optimize first the easy cases of CALL_FUNCTION and move slowly to add more and more calls into it. This approach has the advantage that allow us to take it in small increments.

    15. 5 remaining items

    16. markshannon commented on Oct 18, 2021

      @markshannon
      markshannon
      on Oct 18, 2021
      Member

      New changeset 70945d5 by Mark Shannon in branch 'main':
      bpo-45256: Avoid C calls for most Python to Python calls. (GH-28937)
      70945d5

    17. pablogsal commented on Oct 27, 2021

      @pablogsal
      pablogsal
      on Oct 27, 2021
      MemberAuthor

      Unfortunately, seems that #28937 has broken the AMD64 FreeBSD Shared 3.x buildbot:

      https://buildbot.python.org/all/#/builders/483/builds/1003/steps/5/logs/stdio

      The buildbot was green until we merged this

    18. markshannon commented on Oct 28, 2021

      @markshannon
      markshannon
      on Oct 28, 2021
      Member

      New changeset 7f61d9d by Mark Shannon in branch 'main':
      bpo-45256: Rationalize code around Python-to-Python calls a bit. (GH-29235)
      7f61d9d

    19. markshannon commented on Nov 17, 2021

      @markshannon
      markshannon
      on Nov 17, 2021
      Member

      https://bugs.python.org/issue45829 is the related issue for special methods

    20. markshannon commented on Jan 5, 2022

      @markshannon
      markshannon
      on Jan 5, 2022
      Member

      New changeset 332e6b9 by Brandt Bucher in branch 'main':
      bpo-45256: Don't track the exact depth of each InterpreterFrame (GH-30372)
      332e6b9

    21. markshannon
      closed this as completedon Jan 6, 2022
    22. markshannon
      closed this as completedon Jan 6, 2022
    23. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    24. kumaraditya303
      added a commit that references this issue on Jan 3, 2023

      gh-89419: gdb: fix bug causing `AttributeError` in py-locals when no …

      8586949
    25. eli-schwartz
      added 2 commits that reference this issue on Jan 4, 2023

      pythongh-89419: gdb: fix bug causing AttributeError in py-locals when…

      2953122

      pythongh-89419: gdb: fix bug causing AttributeError in py-locals when…

      d34b3ef
    26. kumaraditya303
      added a commit that references this issue on Jan 4, 2023

      [3.11] gh-89419: gdb: fix bug causing AttributeError in py-locals whe…

      d8073ee
    27. vstinner
      added 3 commits that reference this issue on Aug 22, 2023

      pythongh-108297: Update test_crashers

      e4dced8

      pythongh-108297: Update test_crashers

      00b9633

      pythongh-108297: Update test_crashers

      895f2fa
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.11only security fixes

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions