Repository navigation
API for setting the memory allocator used by Python #47579
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2013-07-07.15:25:49.113>
created_at = <Date 2008-07-09.19:48:51.979>
labels = ['interpreter-core', 'type-feature']
title = 'API for setting the memory allocator used by Python'
updated_at = <Date 2021-03-09.11:39:48.738>
user = 'https://bugs.python.org/jlaurila'bugs.python.org fields:
activity = <Date 2021-03-09.11:39:48.738>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2013-07-07.15:25:49.113>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2008-07-09.19:48:51.979>
creator = 'jlaurila'
dependencies = []
files = ['30451', '30452', '30496', '30537', '30559', '30560', '30753']
hgrepos = []
issue_num = 3329
keywords = ['patch']
message_count = 48.0
messages = ['69482', '69483', '69484', '69494', '69497', '69499', '69511', '78995', '79309', '91957', '142981', '183587', '183590', '183591', '183947', '183950', '183951', '190429', '190528', '190529', '190534', '190539', '190741', '190937', '190940', '190951', '190962', '191029', '191030', '191049', '191050', '191074', '191077', '191165', '191184', '191379', '191436', '191508', '192220', '192504', '192506', '192508', '192509', '192570', '192576', '388351', '388352', '388356']
nosy_count = 18.0
nosy_names = ['barry', 'rhettinger', 'gregory.p.smith', 'jcea', 'amaury.forgeotdarc', 'ncoghlan', 'Rhamphoryncus', 'pitrou', 'kristjan.jonsson', 'vstinner', 'jszakmeister', 'tlesher', 'jlaurila', 'trent', 'neilo', 'pjmcnerney', 'python-dev', 'miss-islington']
pr_nums = ['24795', '24799', '24800']
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue3329'
versions = ['Python 3.4']
Activity
jlaurila commentedon Jul 9, 2008
Currently Python always uses the C library malloc/realloc/free as the
underlying mechanism for requesting memory from the OS, but especially
on memory-limited platforms it is often desirable to be able to override
the allocator and to redirect all Python's allocations to use a special
heap. This will make it possible to free memory back to the operating
system without restarting the process, and to reduce fragmentation by
separating Python's allocations from the rest of the program.
The proposal is to make it possible to set the allocator used by the
Python interpreter by calling the following function before Py_Initialize():
Direct function calls to malloc/realloc/free in obmalloc.c must be
replaced with calls through the function pointers set through this
function. By default these would of course point to the C stdlib
malloc/realloc/free.
brettcannon commentedon Jul 9, 2008
Is registering pointers to functions really necessary, or would defining
macros work as well? From a performance perspective I would like to
avoid having a pointer indirection step every time malloc/realloc/free
is called.
I guess my question becomes, Jukka, is this more for alternative
implementations of Python where changes to source are already expected,
or for apps that embed Python where a change of malloc/realloc/free
varies from app to app that dynamically loads Python?
Rhamphoryncus commentedon Jul 9, 2008
How would this allow you to free all memory? The interpreter will still
reference it, so you'd have to have called Py_Finalize already, and
promise not to call Py_Initialize afterwords. This further supposes the
process will live a long time after killing off the interpreter, but in
that case I recommend putting python in a child process instead.
jlaurila commentedon Jul 10, 2008
Brett, the ability to define the allocator dynamically at runtime could
be a compile time option, turned on by default only on small memory
platforms. On most platforms you can live with plain old malloc and may
want to avoid the indirection. If no other platform is interested in
this, we can just make it a Symbian-specific extension but I wanted to
see if there's general interest in this.
The application would control the lifecycle of the Python heap, and this
seemed like the most natural way for the application to tell the
interpreter which heap instance to use.
Adam, the cleanup would work by freeing the entire heap used by Python
after calling Py_Finalize. In the old PyS60 code we made Python 2.2.2
clean itself completely by freeing the Python-specific heap and making
sure all pointers to heap-allocated items are properly reinitialized.
Yes, there are various static pointers that are initially set to NULL,
initialized to point at things on the heap and not reset to NULL at
Py_Finalize, and these are currently an obstacle to calling
Py_Initialize again. I'm considering submitting a separate ticket about
that since it seems like the ability to free the heap combined with the
ability to reinitialize the static pointers could together make full
cleanup possible.
ncoghlan commentedon Jul 10, 2008
Given where we are in the release cycle, I've bumped the target releases
to 2.7/3.1. So Symbian are probably going to have to do something
port-specific anyway in order to get 2.6/3.0 up and running.
And in terms of hooking into this kind of thing, some simple macros that
can be overriden in pyport.h (as Brett suggested) may be a better idea
than baking any specific approach into the core interpreter.
rhettinger commentedon Jul 10, 2008
I think it is reasonable to get a macro definition change into 2.6.
The OP's request is essential for his application (running Python
on Nokia phones) and it would be a loss to wait two years for this.
Also, his request for a macro will enable another important piece
of functionality -- allowing a build to intercept and instrument all
calls to the memory allocator.
Barry, can you rule on whether to keep this open for consideration in
2.6. It seems daft to postpone this discussion indefinitely. If we
can agree to a simple, non-invasive solution while there is still yet
another beta, then it makes sense to proceed.
Rhamphoryncus commentedon Jul 10, 2008
Basically you just want to kick the malloc implementation into doing
some housekeeping, freeing its caches? I'm kinda surprised you don't
add the hook directly to your libc's malloc.
IMO, there's no use-case for this until Py_Finalize can completely tear
down the interpreter, which requires a lot of special work (killing(!)
daemon threads, unloading C modules, etc), and nobody intends to do that
at this point.
The practical alternative, as I said, is to run python in a subprocess.
Let the OS clean up after us.
neilo commentedon Jan 3, 2009
I'll be in agreement here. I integrated Python into a game engine not
too long ago, and had to a do a fair chunk of work to isolate Python
into it's own heap - given that fragmentation on low memory systems can
be a bit of a killer. Would also make future upgrades a heck of a lot
easier too, as there'd be no need to do a search for all references and
carefully replace them all.
jlaurila commentedon Jan 7, 2009
Brett is right. Macroing the memory allocator is a better choice than
forcing indirection on all platforms. We did this on Python for S60,
using the macros PyCore_{MALLOC,REALLOC,FREE}_FUNC for interpreter's
allocations, and then redirected those to a mechanism that allows to set
the allocator at runtime.
Sorry we don't have a clean patch at present for this change only, but
in case anyone's interested the full source is at
https://garage.maemo.org/frs/?group_id=854
pjmcnerney commentedon Aug 25, 2009
Has the ability to set the memory allocator been added to Python 2.7/3.1?
Thanks,
PJ
pitrou commentedon Aug 25, 2011
All this needs is a patch.
Note that there are some places where we call malloc()/free() without going through our abstraction API. This is not in allocation-heavy paths, though.
24 remaining items
vstinner commentedon Jun 17, 2013
Convert changeset 6661a8154eb3 into a patch: py_setallocators-6.patch.
vstinner commentedon Jun 18, 2013
Update the patch to follow the API described in the PEP-445 (2013-06-18 22:33:41 +0200).
vstinner commentedon Jun 20, 2013
Update patch according to the last version of the PEP.
vstinner commentedon Jul 2, 2013
Updated patch (version 9):
python-dev commentedon Jul 7, 2013
New changeset ca78c974e938 by Victor Stinner in branch 'default':
Issue bpo-3329: Implement the PEP-445
http://hg.python.org/cpython/rev/ca78c974e938
vstinner commentedon Jul 7, 2013
Ok, let see if buildbots like the PEP-445 (keep this issue open until we have the result of all 3.4 buildbots).
I created the issue bpo-18392 to document PyObject_Malloc().
vstinner commentedon Jul 7, 2013
It looks like the changeset ca78c974e938 broke the "x86 XP-4 3.x" buildbot:
buildbot.python.org/all/builders/x86 XP-4 3.x/builds/8795/
python-dev commentedon Jul 7, 2013
New changeset 51ed51d10e60 by Victor Stinner in branch 'default':
Issue bpo-3329: Fix _PyObject_ArenaVirtualFree()
http://hg.python.org/cpython/rev/51ed51d10e60
vstinner commentedon Jul 7, 2013
Buildbots are happy, changeset 51ed51d10e60 fixed the memory leak on Windows XP. Let's close this issue, 5 years after its creation!
kristjanvalur commentedon Jul 7, 2013
Well done.
vstinner commentedon Mar 9, 2021
New changeset 0d6bd1c by Victor Stinner in branch 'master':
bpo-3329: Fix typo in PyObjectArenaAllocator doc (GH-24795)
0d6bd1c
miss-islington commentedon Mar 9, 2021
New changeset 5ca02c4 by Miss Islington (bot) in branch '3.8':
bpo-3329: Fix typo in PyObjectArenaAllocator doc (GH-24795)
5ca02c4
miss-islington commentedon Mar 9, 2021
New changeset ea46c7b by Miss Islington (bot) in branch '3.9':
bpo-3329: Fix typo in PyObjectArenaAllocator doc (GH-24795)
ea46c7b