Skip to content

ProcessPoolExecutor exception memory leak #90943

Closed
Closed
ProcessPoolExecutor exception memory leak#90943
Labels
3.9 (EOL)end of lifeperformancePerformance or resource usagestdlibStandard Library Python modules in the Lib/ directory

Description

@themylogin
themylogin
mannequin
opened on Feb 18, 2022
BPO 46787
Nosy @themylogin
PRs
  • bpo-46787: Fix ProcessPoolExecutor exception memory leak #31408
    		•
    Files
  • 1.py
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2022-02-18.15:16:10.733>
labels = ['library', '3.9', 'performance']
title = 'ProcessPoolExecutor exception memory leak'
updated_at = <Date 2022-02-18.15:23:21.328>
user = 'https://github.com/themylogin'

    bugs.python.org fields:

    activity = <Date 2022-02-18.15:23:21.328>
actor = 'python-dev'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2022-02-18.15:16:10.733>
creator = 'themylogin'
dependencies = []
files = ['50628']
hgrepos = []
issue_num = 46787
keywords = ['patch']
message_count = 1.0
messages = ['413485']
nosy_count = 2.0
nosy_names = ['python-dev', 'themylogin']
pr_nums = ['31408']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue46787'
versions = ['Python 3.9']

    Activity

    1. themylogin commented on Feb 18, 2022

      @themylogin
      themyloginmannequin
      on Feb 18, 2022
      MannequinAuthor

      If an exception occurs in ProcessPoolExecutor work item, all the exception frame local variables are not garbage collected (or are garbage collected too lately) because they are referenced by the exception's traceback.

      Attached file is a test case. With unpatched python 3.9 (debian bullseye) it prints:

      root@truenas[~/freenas/freenas]# python test.py
      At iteration 0 memory usage is 226070528
      At iteration 1 memory usage is 318763008
      At iteration 2 memory usage is 318509056
      At iteration 3 memory usage is 321662976
      At iteration 4 memory usage is 321404928
      At iteration 5 memory usage is 324292608
      At iteration 6 memory usage is 324296704
      At iteration 7 memory usage is 326922240
      At iteration 8 memory usage is 326922240
      At iteration 9 memory usage is 329543680

      With the proposed patch there is no memory usage growth:

      At iteration 0 memory usage is 226410496
      At iteration 1 memory usage is 226451456
      At iteration 2 memory usage is 226451456
      At iteration 3 memory usage is 226443264
      At iteration 4 memory usage is 226443264
      At iteration 5 memory usage is 226435072
      At iteration 6 memory usage is 226426880
      At iteration 7 memory usage is 226426880
      At iteration 8 memory usage is 226435072
      At iteration 9 memory usage is 226426880

    2. themylogin
      added
      3.9 (EOL)end of life
      stdlibStandard Library Python modules in the Lib/ directory
      performancePerformance or resource usage
      on Feb 18, 2022
    3. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    4. kumaraditya303
      closed this as completedon May 6, 2022

    5. clchiou commented on Sep 25, 2022

      @clchiou
      clchiou
      on Sep 25, 2022

      I wonder if this patch also needs to be applied to Lib/multiprocessing/pool.py as it appears to me that Lib/concurrent/futures/process.py and Lib/multiprocessing/pool.py are quite similar in this aspect: sending exceptions with (stringified) traceback across process boundary. What do you think? @themylogin

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.9 (EOL)end of lifeperformancePerformance or resource usagestdlibStandard Library Python modules in the Lib/ directory

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions