Skip to content

Add PYTHONMALLOC env var and add support for malloc debug hooks in release mode #70703

Closed
Closed
Add PYTHONMALLOC env var and add support for malloc debug hooks in release mode#70703
Labels
type-featureA feature request or enhancement

Description

@vstinner
vstinner
opened on Mar 9, 2016
BPO 26516
Nosy @malemburg, @vstinner, @serhiy-storchaka
Files
  • pymem.patch
  • pymem-2.patch
  • pymem-3.patch
  • pymem-4.patch
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2016-03-17.00:41:28.514>
created_at = <Date 2016-03-09.11:15:15.050>
labels = ['type-feature']
title = 'Add PYTHONMALLOC env var and add support for malloc debug hooks in release mode'
updated_at = <Date 2016-04-19.15:03:12.525>
user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2016-04-19.15:03:12.525>
actor = 'python-dev'
assignee = 'none'
closed = True
closed_date = <Date 2016-03-17.00:41:28.514>
closer = 'vstinner'
components = []
creation = <Date 2016-03-09.11:15:15.050>
creator = 'vstinner'
dependencies = []
files = ['42098', '42102', '42104', '42119']
hgrepos = []
issue_num = 26516
keywords = ['patch']
message_count = 19.0
messages = ['261413', '261414', '261415', '261416', '261425', '261428', '261429', '261432', '261514', '261746', '261747', '261748', '261761', '261765', '261769', '261777', '261872', '261955', '263746']
nosy_count = 4.0
nosy_names = ['lemburg', 'vstinner', 'python-dev', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'patch review'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue26516'
versions = ['Python 3.6']

    Activity

    1. vstinner commented on Mar 9, 2016

      @vstinner
      vstinner
      on Mar 9, 2016
      MemberAuthor

      Attached patch:

      • Add PYTHONMALLOC env var which accepts 4 values:

        • pymalloc: use pymalloc for PyObject_Malloc(), malloc for PyMem_Malloc() and PyMem_RawMalloc()
        • pymalloc_debug: pymalloc + debug hooks
        • malloc: use malloc for PyObject_Malloc(), PyMem_Malloc() and PyMem_RawMalloc()
        • malloc_debug: malloc + debug hooks

      • Add support for debug hooks in release mode

      • Add unit test for debug hooks in test_capi.py

      • Add unit on misuse of memory allocators in test_capi.py

      PYTHONMALLOC is used even if -E command line option is used, I prefer to keep the code simple. PYTHONMALLOC must be checked before the first call to any Python memory allocator, so it must occur very earlier. Well, it's simply the first instruction of main()...

      My main use case is to be able to use debug hooks to detect:

      • misuse of python memory allocators like object allocated with PyObject_Malloc() and freed with PyMem_Free()
      • buffer overflow
      • buffer underlow
    2. vstinner
      added
      type-featureA feature request or enhancement
      on Mar 9, 2016

    3. vstinner commented on Mar 9, 2016

      @vstinner
      vstinner
      on Mar 9, 2016
      MemberAuthor

      The output of sys._debugmallocstats() contains different info:

      • Stats of type free lists
      • pymalloc stats
      • number of calls to memory allocators

      The available info depends on:

      • pymalloc disabled at compilation with ./configure --without-pymalloc
      • pymalloc disabled at runtime with PYTHONMALLOC=malloc or PYTHONMALLOC=malloc_debug
      • debug hooks adds the "times object malloc called" counter: by default if Python is compiled with ./configure --with-pydebug, or enabled at runtime using PYTHONMALLOC=pymalloc_debug or PYTHONMALLOC=malloc_debug

    4. vstinner commented on Mar 9, 2016

      @vstinner
      vstinner
      on Mar 9, 2016
      MemberAuthor

      Example with Python compiled in release mode.

      By default, a buffer overflow is not detected. It may crash later, in a random place...

      $ ./python -c 'import _testcapi; _testcapi.pymem_buffer_overflow()'

      Enabling debug hooks detects the buffer overflow immediatly:

      $ PYTHONMALLOC=pymalloc_debug ./python -c 'import _testcapi; _testcapi.pymem_buffer_overflow()'
Debug memory block at address p=0x1a7f490: API 'm'
    16 bytes originally requested
    The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected.
    The 8 pad bytes at tail=0x1a7f4a0 are not all FORBIDDENBYTE (0xfb):
        at tail+0: 0x78 *** OUCH
        at tail+1: 0xfb
        at tail+2: 0xfb
        at tail+3: 0xfb
        at tail+4: 0xfb
        at tail+5: 0xfb
        at tail+6: 0xfb
        at tail+7: 0xfb
    The block was made by call python/issues-test-cpython#35014 to debug malloc/realloc.
    Data at p: cb cb cb cb cb cb cb cb cb cb cb cb cb cb cb cb
Fatal Python error: bad trailing pad byte

      Current thread 0x00007fca30572700 (most recent call first):
      File "<string>", line 1 in <module>
      Abandon (core dumped)

    5. vstinner commented on Mar 9, 2016

      @vstinner
      vstinner
      on Mar 9, 2016
      MemberAuthor

      The motivation to support PYTHONMALLOC=malloc (always use malloc) and not just PYTHONMALLOC=debug (enable debug hooks) is to allow to use external memory debugger like Valgrind.

      Valgrind doesn't like pymalloc (pymalloc raises false alarms), we had to a configuration option for it: ./configure --with-valgrind.

      Using PYTHONMALLOC=malloc allows to use Valgrind on the system Python, it's easier to use. No need to recompile Python.

    6. python-dev commented on Mar 9, 2016

      @python-dev
      python-devmannequin
      on Mar 9, 2016
      Mannequin

      New changeset af2fc10f703b by Victor Stinner in branch '3.5':
      Issue bpo-26516: Enhance Python mem allocators doc
      https://hg.python.org/cpython/rev/af2fc10f703b

    7. vstinner commented on Mar 9, 2016

      @vstinner
      vstinner
      on Mar 9, 2016
      MemberAuthor

      More complete patch (version 2):

      • Document PYTHONMALLOC environment variable
      • Add PYTHONMALLOC=debug to install debug hooks without forcing a specific memory allocator (keep the default memory allocator)
      • Fix sys._debugmallocstats() for PYTHONMALLOC=malloc: don't display pymalloc stats
      • _testcapi.pymem_api_misuse(): more realistic code, use PyMem_Mallloc() + PyMem_RawFree(). This code works in release mode (since PyMem and PyMem_Raw use the same allocator: malloc/free), but fail with a fatal error with debug hooks (API misused).

    8. vstinner commented on Mar 9, 2016

      @vstinner
      vstinner
      on Mar 9, 2016
      MemberAuthor

      When I wrote the PEP-445, I already proposed to add a new environment variable to install debug hooks on a Python compiled in release mode:

      https://www.python.org/dev/peps/pep-0445/#add-a-new-pydebugmalloc-environment-variable

      But the proposition was rejected because of the PEP-432, but 3 years later, this PEP is still stuck at the draft state. So I propose to move on.

      Anyway, PYTHONMALLOC is really a corner case since it must be read *very early* (first instruction of main()).

      Oh by the way, I forgot to explain the initial motivation for this change! It's the issue bpo-26249: "Change PyMem_Malloc to use pymalloc allocator". With this change, applications misuing PyMem_Malloc API (ex: alloc memory with PyMem_Malloc(), free memory with free()) will start to crash. PYTHONMALLOC=debug should help these applications to detect bugs easier and get an obvious error message.

    9. vstinner commented on Mar 9, 2016

      @vstinner
      vstinner
      on Mar 9, 2016
      MemberAuthor

      Patch 3:

      • Ooops, I updated pymem_api_misuse(), but I forgot to update the related unit test. It's now fixed.

    10. vstinner commented on Mar 10, 2016

      @vstinner
      vstinner
      on Mar 10, 2016
      MemberAuthor

      Oh, I forgot to update wmain() for Windows.

      Patch 4 handles wmain() too. I added a few comments and reformatted some parts of the code.

    11. python-dev commented on Mar 14, 2016

      @python-dev
      python-devmannequin
      on Mar 14, 2016
      Mannequin

      New changeset aa280432e9c7 by Victor Stinner in branch 'default':
      Add PYTHONMALLOC env var
      https://hg.python.org/cpython/rev/aa280432e9c7

    12. vstinner commented on Mar 14, 2016

      @vstinner
      vstinner
      on Mar 14, 2016
      MemberAuthor

      I reworked my patch before pushing it:

      • it now respects -E and -I command line options
      • I documented changes in Doc/, Misc/NEWS, What's New in Python 3.6, Misc/README.valgrind, etc.
      • Debug hooks are now also installed when Python is compiled in debug mode without pymalloc
      • PYTHONMALLOCSTATS is now ignored if the allocator if pymalloc is not used
      • PYTHONMALLOC is ignored if it's an empty string

    13. vstinner commented on Mar 14, 2016

      @vstinner
      vstinner
      on Mar 14, 2016
      MemberAuthor

      Note: I just checked PYTHONMALLOCSTATS=1 with -X tracemalloc. It looks like it works as expected. Tracemalloc hooks are unregistered before stats are displayed at exit, _Pymem_PymallocEnabled() returns 1 as expected.

    14. vstinner commented on Mar 14, 2016

      @vstinner
      vstinner
      on Mar 14, 2016
      MemberAuthor

      test_capi fails on Windows:
      http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/7348/steps/test/logs/stdio

      Probably a issue with newline characters.

    15. python-dev commented on Mar 14, 2016

      @python-dev
      python-devmannequin
      on Mar 14, 2016
      Mannequin

      New changeset 3c3df33f2655 by Victor Stinner in branch 'default':
      Issue bpo-26516: Fix test_capi on Windows
      https://hg.python.org/cpython/rev/3c3df33f2655

    16. python-dev commented on Mar 14, 2016

      @python-dev
      python-devmannequin
      on Mar 14, 2016
      Mannequin

      New changeset d8d6ec1333e6 by Victor Stinner in branch 'default':
      Issue bpo-26516: Fix test_capi on 32-bit system
      https://hg.python.org/cpython/rev/d8d6ec1333e6

    17. python-dev commented on Mar 14, 2016

      @python-dev
      python-devmannequin
      on Mar 14, 2016
      Mannequin

      New changeset 7534eb7bd57e by Victor Stinner in branch 'default':
      Issue bpo-26516: Fix test_capi on AIX
      https://hg.python.org/cpython/rev/7534eb7bd57e

    18. vstinner commented on Mar 17, 2016

      @vstinner
      vstinner
      on Mar 17, 2016
      MemberAuthor

      Buildbots are happy, I close the issue.

    19. vstinner
      closed this as completedon Mar 17, 2016

    20. python-dev commented on Mar 18, 2016

      @python-dev
      python-devmannequin
      on Mar 18, 2016
      Mannequin

      New changeset 7b079adb0774 by Victor Stinner in branch 'default':
      Enhance documentation on malloc debug hooks
      https://hg.python.org/cpython/rev/7b079adb0774

    21. python-dev commented on Apr 19, 2016

      @python-dev
      python-devmannequin
      on Apr 19, 2016
      Mannequin

      New changeset c4c14e34e528 by Victor Stinner in branch 'default':
      Don't define _PyMem_PymallocEnabled() if pymalloc is disabled
      https://hg.python.org/cpython/rev/c4c14e34e528

    22. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      type-featureA feature request or enhancement

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions