Skip to content

Custom pickler memory leak #89988

Closed
Closed
Custom pickler memory leak#89988
Labels
3.10 (EOL)end of life3.11only security fixes3.12only security fixesperformancePerformance or resource usagestdlibStandard Library Python modules in the Lib/ directory

Description

@douglas-raillard-arm
douglas-raillard-arm
mannequin
opened on Nov 17, 2021
BPO 45830
Nosy @douglas-raillard-arm

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-11-17.16:42:28.795>
labels = ['library', '3.10', 'performance']
title = 'Custom pickler memory leak'
updated_at = <Date 2021-11-17.16:42:28.795>
user = 'https://github.com/douglas-raillard-arm'

bugs.python.org fields:

activity = <Date 2021-11-17.16:42:28.795>
actor = 'douglas-raillard-arm'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-11-17.16:42:28.795>
creator = 'douglas-raillard-arm'
dependencies = []
files = []
hgrepos = []
issue_num = 45830
keywords = []
message_count = 1.0
messages = ['406478']
nosy_count = 1.0
nosy_names = ['douglas-raillard-arm']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue45830'
versions = ['Python 3.10']

Activity

  1. douglas-raillard-arm commented on Nov 17, 2021

    @douglas-raillard-arm
    douglas-raillard-armmannequin
    on Nov 17, 2021
    MannequinAuthor

    The following script exhibits the memory leak. It only happens if "dispatch_table" is set _before_ calling super().__init__, which is pretty unexpected.

        import pickle
    import io
    import gc
    import tracemalloc

    tracemalloc.start(10)
    snap = tracemalloc.take_snapshot()

    class MyPickler(pickle.Pickler):
        def __init__(self, *args, **kwargs):
            # Swapping the next 2 lines "solves" the memory leak for some reason
            self.dispatch_table = dict()
            super().__init__(*args, **kwargs)

    l=[]
    for i in range(10000):
        if i % 1000 == 0:
            print('='*80)
            snap2 = tracemalloc.take_snapshot()
            stats=snap2.compare_to(snap, 'lineno')
            for s in stats[:10]:
                print(s)
            snap = snap2

        f = io.BytesIO()
        MyPickler(f)
        gc.collect()

    The output of the last iteration is as follow. The leak of 562 kiB is apparent:

    testmem.py:12: size=562 KiB (+62.5 KiB), count=9000 (+1000), average=64 B
/usr/lib/python3.10/tracemalloc.py:125: size=2376 B (-72 B), count=33 (-1), average=72 B
/usr/lib/python3.10/tracemalloc.py:129: size=72 B (+72 B), count=1 (+1), average=72 B
/usr/lib/python3.10/tracemalloc.py:502: size=252 B (+28 B), count=9 (+1), average=28 B
/usr/lib/python3.10/tracemalloc.py:498: size=2104 B (+0 B), count=36 (+0), average=58 B
/home/dourai01/Work/lisa/lisa/testmem.py:10: size=1844 B (+0 B), count=9 (+0), average=205 B
/usr/lib/python3.10/tracemalloc.py:193: size=1680 B (+0 B), count=35 (+0), average=48 B
/usr/lib/python3.10/tracemalloc.py:547: size=1256 B (+0 B), count=3 (+0), average=419 B
/usr/lib/python3.10/tracemalloc.py:226: size=832 B (+0 B), count=2 (+0), average=416 B
/usr/lib/python3.10/tracemalloc.py:173: size=800 B (+0 B), count=2 (+0), average=400 B

    If "dispatch_table" is set after calling super().__init__, there is no leak anymore:

    /usr/lib/python3.10/tracemalloc.py:135: size=740 B (+740 B), count=7 (+7), average=106 B
/usr/lib/python3.10/tracemalloc.py:125: size=2088 B (-656 B), count=29 (-4), average=72 B
/usr/lib/python3.10/tracemalloc.py:136: size=320 B (+320 B), count=1 (+1), average=320 B
/usr/lib/python3.10/tracemalloc.py:132: size=0 B (-256 B), count=0 (-1)
/usr/lib/python3.10/tracemalloc.py:129: size=72 B (+72 B), count=1 (+1), average=72 B
/usr/lib/python3.10/tracemalloc.py:498: size=2008 B (+48 B), count=34 (+1), average=59 B
/usr/lib/python3.10/tracemalloc.py:193: size=1584 B (+48 B), count=33 (+1), average=48 B
/usr/lib/python3.10/tracemalloc.py:502: size=196 B (-28 B), count=7 (-1), average=28 B
/usr/lib/python3.10/tracemalloc.py:126: size=84 B (+28 B), count=3 (+1), average=28 B
  2. douglas-raillard-arm
    added
    3.10 (EOL)end of life
    stdlibStandard Library Python modules in the Lib/ directory
    performancePerformance or resource usage
    on Nov 17, 2021
  3. ezio-melotti
    transferred this issue fromon Apr 10, 2022
  4. serhiy-storchaka
    added this to Pickle and copy issues 🥒on May 19, 2022

  5. tigerinus commented on Jun 3, 2022

    @tigerinus
    tigerinus
    on Jun 3, 2022

    I can verify this also happens on Python 3.9.2

    I also have a demonstration of this memory leak at https://github.com/tigerinus/pickle_memory_leak_with_msgspec

  6. tigerinus commented on Jun 4, 2022

    @tigerinus
    tigerinus
    on Jun 4, 2022

    I can verify this also happens on Python 3.9.2

    I also have a demonstration of this memory leak at https://github.com/tigerinus/pickle_memory_leak_with_msgspec

    The issue I found turns out to be an issue of msgspec.Struct and should be fixed by msgspec/msgspec#117

  7. kumaraditya303
    added
    3.11only security fixes
    3.12only security fixes
    on Jun 27, 2022
  8. serhiy-storchaka
    added a commit that references this issue on Jun 28, 2022

    GH-89988: Fix memory leak in pickle.Pickler dispatch_table lookup (GH…

    01ef1f9
  9. miss-islington
    added a commit that references this issue on Jun 28, 2022

    pythonGH-89988: Fix memory leak in pickle.Pickler dispatch_table look…

    d771639
  10. kumaraditya303
    added a commit that references this issue on Jun 28, 2022

    pythonGH-89988: Fix memory leak in pickle.Pickler dispatch_table look…

    ccaa5ea
  11. pablogsal
    added a commit that references this issue on Jun 28, 2022

    [3.10] GH-89988: Fix memory leak in pickle.Pickler dispatch_table loo…

    079ea44
  12. gvanrossum
    added a commit that references this issue on Jun 30, 2022

    pythonGH-89988: Fix memory leak in pickle.Pickler dispatch_table look…

    8699848
  13. miss-islington
    added a commit that references this issue on Jul 14, 2022

    GH-89988: Fix memory leak in pickle.Pickler dispatch_table lookup (GH…

    91f7914
  14. serhiy-storchaka
    closed this as completedon Jul 14, 2022
  15. serhiy-storchaka
    moved this to Done in Pickle and copy issues 🥒on Jul 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.10 (EOL)end of life3.11only security fixes3.12only security fixesperformancePerformance or resource usagestdlibStandard Library Python modules in the Lib/ directory

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions