Repository navigation
Use VirtualAlloc to allocate memory arenas #57692
Description
| BPO | 13483 |
|---|---|
| Nosy | @loewis, @pitrou, @vstinner, @giampaolo, @tjguk, @tpn, @briancurtin |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2013-06-27.10:24:52.456>
created_at = <Date 2011-11-26.13:12:42.221>
labels = ['interpreter-core', 'OS-windows', 'performance']
title = 'Use VirtualAlloc to allocate memory arenas'
updated_at = <Date 2013-06-27.10:24:52.456>
user = 'https://github.com/pitrou'bugs.python.org fields:
activity = <Date 2013-06-27.10:24:52.456>
actor = 'loewis'
assignee = 'none'
closed = True
closed_date = <Date 2013-06-27.10:24:52.456>
closer = 'loewis'
components = ['Interpreter Core', 'Windows']
creation = <Date 2011-11-26.13:12:42.221>
creator = 'pitrou'
dependencies = []
files = ['23784', '26069', '26070']
hgrepos = []
issue_num = 13483
keywords = ['patch']
message_count = 23.0
messages = ['148399', '148605', '148611', '148612', '148621', '148623', '148625', '163350', '163351', '189760', '189771', '189824', '189825', '189850', '189876', '190427', '191252', '191253', '191461', '191504', '191506', '191507', '191939']
nosy_count = 9.0
nosy_names = ['loewis', 'pitrou', 'vstinner', 'giampaolo.rodola', 'tim.golden', 'trent', 'brian.curtin', 'neologix', 'python-dev']
pr_nums = []
priority = 'low'
resolution = 'fixed'
stage = 'commit review'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue13483'
versions = ['Python 3.4']
Activity
pitrou commentedon Nov 26, 2011
Similar to issue bpo-11849, this patch proposes to use VirtualAlloc/VirtualFree to allocate the Python allocator's memory arenas (rather than malloc() / free()). It might help release more memory if there is some fragmentation, although I don't know how Microsoft's malloc() works.
loewis commentedon Nov 29, 2011
The patch looks good to me.
To study Microsoft's malloc, see VC\crt\src\malloc.c. Typically, it uses HeapAlloc from the CRT heap, unless it's in 32-bit mode, and __active_heap is either __V6_HEAP or __V5_HEAP. This is determined at startup by __heap_select, inspecting an environment variable __MSVCRT_HEAP_SELECT. If that's not set, the CRT heap is used.
The CRT heap, in turn, is created with HeapCreate (no flags).
As an alternative approach, Python could consider completely dropping obmalloc on Windows, and using a Windows Low Fragementation Heap (LFH) instead, with HEAP_NO_SERIALIZE (as the heap would be protected by the GIL).
If we take the route proposed by this patch, I recommend also dropping all other CRT malloc() calls in Python, and make allocations from the process heap instead (that's a separate issue, though).
pitrou commentedon Nov 29, 2011
Ah, right, I guessed it was using HeapAlloc indeed. What would be more
interesting is how HeapAlloc works :)
I think it would be nice to know whether the patch has a chance of being
useful before committing it. I did it as a thought experiment after the
similar change was committed for Unix, but I'm not an expert in Windows
internals. Perhaps HeapAlloc deals fine with fragmentation? Tim, Brian,
do you know anything about this?
I'm not sure that would serve the same purpose as obmalloc, which
(AFAIU) is very fast at the expense of compacity.
tjguk commentedon Nov 29, 2011
'fraid not. I've never had to dig into the allocation stuff at this level.
loewis commentedon Nov 29, 2011
Unfortunately, the implementation of HeapAlloc isn't really documented.
If Reactos is right, it looks like this: http://bit.ly/t2NPHh
Blocks < 1024 bytes are allocated from per-size free lists.
Blocks < Heap->VirtualMemoryThreshold are allocated through the free
list for variable-sized blocks of the heap.
Other blocks are allocated through ZwAllocateVirtualMemory, adding
sizeof(HEAP_VIRTUAL_ALLOC_ENTRY) in the beginning. I think this header
will cause malloc() to allocate one extra page in front of an arena.
I'd expect that LFH heaps are also very fast. The major difference I can
see is that blocks in the LFH heap still have an 8-byte header (possibly
more on a 64-bit system). So I wouldn't expect any speed savings, but
(possibly relevant) memory savings from obmalloc.
briancurtin commentedon Nov 29, 2011
Unfortunately, no. It's on my todo list of things to understand but I don't see that happening in the near future.
I'm willing to run tests or benchmarks for this issue, but that's likely the most I can provide.
pitrou commentedon Nov 29, 2011
Le mardi 29 novembre 2011 à 22:39 +0000, Brian Curtin a écrit :
Benchmarks would be nice indeed.
loewis commentedon Jun 21, 2012
Here is a benchmark. Based on my assumption that this patch may reduce allocation overheads due to minimizing padding+fragmentation, it allocates a lot of memory, and then waits 20s so you can check in the process explorer what the "Commit Size" of the process is.
For the current 3.3 tree, in 32-bit mode, on a 64-bit Windows 7 installation, I get 464,756K for the unpatched version, and 450,436K for the patched version.
This is a 3% saving, which seems good enough for me.
loewis commentedon Jun 21, 2012
Here is an updated patch.
neologix commentedon May 21, 2013
Martin, do you think your latest patch can be committed?
loewis commentedon May 21, 2013
Antoine's request for benchmarks still stands. I continue to think that it should be applied even in absence of benchmarks. In the absence of third opinions on this specific aspect, I don't think it can be applied.
neologix commentedon May 22, 2013
I can't speak for Antoine, but I guess that the result of pybench
would be enough to make sure it doesn't introduce any regression
(which would be *really* suprising).
As for the memory savings, the benchmark you posted earlier is
conclusive enough IMO (especially since the it can be difficult to
come up with a scheme leading to heap fragmentation).
pitrou commentedon May 22, 2013
I asked for benchmarks because I don't know anything about Windows virtual memory management, but if other people think this patch should go in then it's fine.
The main point of using VirtualAlloc/VirtualFree was, in my mind, to allow *releasing* memory in more cases than when relying on free() (assuming Windows uses some sbrk() equivalent). But perhaps Windows is already tuned to release memory on most free() calls.
loewis commentedon May 23, 2013
Ah ok. I guess tuples.py then indeed demonstrates a saving. I'll apply the patch.
vstinner commentedon May 23, 2013
Set also issue bpo-3329 which proposes an API to define memory allocators.
vstinner commentedon May 31, 2013
I tested VirtualAlloc/VirtualFree versus malloc/free on Windows 7 SP1 64-bit. On my small test, when using VirtualAlloc/VirtualFree, the memory peak is lower (ex: 58.1 MB vs 59.0), and the memory usage is the same or sometimes lower. The difference is small, malloc() implementation on Windows 7 is efficient! But I am in favor of using VirtualAlloc/VirtualFree because it is the native API and the gain may be bigger on a real application.
--
I used the following script for my test:
https://bitbucket.org/haypo/misc/raw/98eb42a3ed2144141d62c75e3d07933839fe2a0c/python/python_memleak.py
I reused get_process_mem_info() code from psutil to get current and peak memory usage (I failed to install psutil, I don't understand why).
I also replace func() of my script with tuples.py to create many tuples.
--
Python < 3.3 wastes a lot of memory with python_memleak.py. Python 3.3 behaves much better thanks to the usage of mmap() on Linux, and the fixed threshold on 64-bit (min=512 bytes, instead of 256).
vstinner commentedon Jun 16, 2013
Martin von Loewis: "If we take the route proposed by this patch, I recommend also dropping all other CRT malloc() calls in Python, and make allocations from the process heap instead (that's a separate issue, though)."
=> see issue bpo-18203
vstinner commentedon Jun 16, 2013
haypo> I tested VirtualAlloc/VirtualFree versus malloc/free
haypo> on Windows 7 SP1 64-bit. On my small test, ...
I realized that I was no precise: I tried attached va.diff patch. I didn't try to replace completly malloc().
vstinner commentedon Jun 19, 2013
According to my test, the memory usage is a little bit better with the patch. So Martin:,do you plan to commit the patch?
Or is a benchmark required? Or should check first check the Low Fragmentation Allocator?
I plan to test the Low Fragmentation Allocator, at least on Windows 7. But I prefer to do it later, I'm working on the PEP-445 right now.
pitrou commentedon Jun 20, 2013
I don't think it can be any better than raw mmap() / VirtualAlloc()...
vstinner commentedon Jun 20, 2013
I mean using the Low Fragmentation Allocator for PyObject_Malloc()
instead of pymalloc.
Martin wrote (msg148605):
"As an alternative approach, Python could consider completely dropping
obmalloc on Windows, and using a Windows Low Fragementation Heap (LFH)
instead, with HEAP_NO_SERIALIZE (as the heap would be protected by the
GIL)."
loewis commentedon Jun 20, 2013
Ok, I'm going to commit this patch. Any further revisions (including reversions) can be done then.
python-dev commentedon Jun 27, 2013
New changeset 44f455e6163d by Martin v. Löwis in branch 'default':
Issue bpo-13483: Use VirtualAlloc in obmalloc on Windows.
http://hg.python.org/cpython/rev/44f455e6163d