Repository navigation
Add a new tracemalloc module to trace memory allocations #63074
Description
| BPO | 18874 |
|---|---|
| Nosy | @tim-one, @rhettinger, @jcea, @amauryfa, @abalkin, @vstinner, @ericsnowcurrently, @JimJJewett, @vajrasky |
| Dependencies | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2013-11-26.12:59:26.321>
created_at = <Date 2013-08-28.23:53:11.383>
labels = ['type-feature']
title = 'Add a new tracemalloc module to trace memory allocations'
updated_at = <Date 2013-12-10.17:49:41.675>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2013-12-10.17:49:41.675>
actor = 'jcea'
assignee = 'none'
closed = True
closed_date = <Date 2013-11-26.12:59:26.321>
closer = 'vstinner'
components = []
creation = <Date 2013-08-28.23:53:11.383>
creator = 'vstinner'
dependencies = ['16742']
files = ['32555', '32724', '32734', '32753', '32789', '32828']
hgrepos = ['206']
issue_num = 18874
keywords = ['patch']
message_count = 57.0
messages = ['196436', '196437', '196438', '196439', '196474', '196484', '196525', '196720', '196879', '197951', '197952', '197954', '198765', '198767', '199039', '199040', '199041', '199085', '199086', '199123', '199386', '200341', '202024', '202438', '202492', '203343', '203485', '203508', '203534', '203536', '203587', '203589', '203615', '203616', '203620', '203621', '203624', '203659', '203669', '203670', '203676', '203815', '203862', '203947', '203988', '203990', '204297', '204309', '204310', '204431', '204432', '204435', '204444', '204472', '204618', '204619', '204622']
nosy_count = 11.0
nosy_names = ['tim.peters', 'rhettinger', 'jcea', 'amaury.forgeotdarc', 'belopolsky', 'vstinner', 'neologix', 'python-dev', 'eric.snow', 'Jim.Jewett', 'vajrasky']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue18874'
versions = ['Python 3.4']
Activity
vstinner commentedon Aug 28, 2013
Thanks to the PEP-445, it becomes possible to trace easily memory allocations. I propose to add a new tracemalloc module computing the memory usage per file and per line number.
I implemented the module as a third party module for Python 2.5-3.4, but it requires to patch and recompile Python:
https://pypi.python.org/pypi/pytracemalloc
My proposed implementation is different:
use a simple C implementation of an hash table called "cfuhash" (coming from the libcfu project) instead of depending on the glib library ; I simplified and adapted the implementation for my usage
no enable() / disable() function: tracemalloc can only be enabled before startup by setting PYTHONTRACEMALLOC=1 environment variable
traces (size of the memory block, Python filename, Python line number) are stored directly in the memory block, not in a separated hash table
I chose PYTHONTRACEMALLOC env var instead of enable()/disable() functions to be able to really trace *all* memory allocated by Python, especially memory allocated at startup, during Python initialization.
TODO:
The (high-level) API should be reviewed and discussed
The documention should be improved
PyMem_Raw API is not hooked yet because the code is not thread-safe (it relies on the Python GIL)
Filenames should not be encoded: the hash table should directly use Unicode strings (PyObject*), encoding a filename allocates memory and may call the garbage collector
Memory is not released at exit
For the documentation, see the following page:
https://pypi.python.org/pypi/pytracemalloc
See also the https://bitbucket.org/haypo/pyfailmalloc project and the issue bpo-18408.
vstinner commentedon Aug 29, 2013
Example of output:
$ PYTHONTRACEMALLOC=1 ./python -m test ... == CPython 3.4.0a1+ (default:2ce9e5f6b47c+, Aug 29 2013, 02:03:02) [GCC 4.7.2 20121109 (Red Hat 4.7.2-8)] == Linux-3.9.4-200.fc18.x86_64-x86_64-with-fedora-18-Spherical_Cow little-endian == /home/haypo/prog/python/tracemalloc/build/test_python_11087 ... [ 1/380] test_grammar [ 2/380] test_opcodes [ 3/380] test_dict [ 4/380] test_builtin [ 5/380] test_exceptions [ 6/380] test_types [ 7/380] test_unittest2013-08-29 02:06:22: Top 25 allocations per file and line
#1: <frozen importlib._bootstrap>:704: size=5 MiB, count=56227, average=105 B
#2: .../tracemalloc/Lib/linecache.py:127: size=1004 KiB, count=8706, average=118 B
#3: .../Lib/unittest/mock.py:1764: size=895 KiB, count=7841, average=116 B
#4: .../Lib/unittest/mock.py:1805: size=817 KiB, count=15101, average=55 B
#5: .../Lib/test/test_dict.py:35: size=768 KiB, count=8, average=96 KiB
#6: <frozen importlib._bootstrap>:274: size=703 KiB, count=4604, average=156 B
#7: ???:?: size=511 KiB, count=4445, average=117 B
#8: .../Lib/unittest/mock.py:350: size=370 KiB, count=1227, average=308 B
#9: .../Lib/unittest/case.py:306: size=343 KiB, count=1390, average=253 B
#10: .../Lib/unittest/case.py:496: size=330 KiB, count=650, average=521 B
#11: .../Lib/unittest/case.py:327: size=291 KiB, count=717, average=416 B
#12: .../Lib/collections/init.py:368: size=239 KiB, count=2170, average=113 B
#13: .../Lib/test/test_grammar.py:132: size=195 KiB, count=1250, average=159 B
#14: .../Lib/unittest/mock.py:379: size=118 KiB, count=152, average=800 B
#15: .../tracemalloc/Lib/contextlib.py:37: size=102 KiB, count=672, average=156 B
#16: <frozen importlib._bootstrap>:1430: size=91 KiB, count=1193, average=78 B
#17: .../tracemalloc/Lib/inspect.py:1399: size=79 KiB, count=104, average=784 B
#18: .../tracemalloc/Lib/abc.py:133: size=77 KiB, count=275, average=289 B
#19: .../Lib/unittest/case.py:43: size=73 KiB, count=593, average=127 B
#20: .../Lib/unittest/mock.py:491: size=67 KiB, count=153, average=450 B
#21: <frozen importlib._bootstrap>:1438: size=64 KiB, count=20, average=3321 B
#22: .../Lib/unittest/case.py:535: size=56 KiB, count=76, average=766 B
#23: .../tracemalloc/Lib/sre_compile.py:508: size=54 KiB, count=115, average=485 B
#24: .../Lib/unittest/case.py:300: size=48 KiB, count=616, average=80 B
#25: .../Lib/test/regrtest.py:1207: size=48 KiB, count=2, average=24 KiB
7333 more: size=4991 KiB, count=28051, average=182 B
Total Python memory: size=17 MiB, count=136358, average=136 B
Total process memory: size=42 MiB (ignore tracemalloc: 22 KiB)
vstinner commentedon Aug 29, 2013
tracemalloc.get_process_memory() should be moved to the shutil module.
vstinner commentedon Aug 29, 2013
I'm not sure that the DisplayGarbage class is still useful. The PEP-442 has been implemented in Python 3.4, so objects with __del__() method are no more "uncollectable". I don't know which kind of objects can enter the gc.garbage list (by default, not using gc.set_debug(gc.DEBUG_SAVEALL)).
amauryfa commentedon Aug 29, 2013
Is it really impossible to use a standard Python dict instead of cfuhash?
vstinner commentedon Aug 29, 2013
I don't know if it is possible or not, I didn't try. Using PyDict has many requirements:
The implementation is not done yet. I would like to be able to compute the size of memory allocated by the _tracemalloc module: size of hash tables.
vstinner commentedon Aug 30, 2013
Ok, let's start with a first patch. It works in the common cases, because they are some corner cases like subinterpreter which might still crash.
The hook on PyMem_RawMalloc() takes the GIL. It is disabled because it has still bugs (it introduces a deadlock in some corner cases!)
vstinner commentedon Sep 1, 2013
New patch (version 2):
TODO:
test_capi deadlock with TRACE_RAW_MALLOC:
vstinner commentedon Sep 3, 2013
Oh, another TODO: drop the (optional) dependency to psutil. Implement get_process_memory() in C (in the _tracemalloc module).
vstinner commentedon Sep 16, 2013
Updated patch.
vstinner commentedon Sep 16, 2013
Include/tracemalloc.h: PyTraceMalloc_DisableTemporary() and PyTraceMalloc_RestoreTemporary() should be removed, they were tests to try to fix issues with subinterpreter when tracing PyMem_RawMalloc().
Lib/test/regrtest.py: These changes should not be commited, they are just here to create easily snapshot files (./python -X tracemalloc -m test).
vstinner commentedon Sep 16, 2013
TODO list:
See also the TODO.txt file in the tracemalloc repository.
vstinner commentedon Oct 1, 2013
The issue bpo-16742 must be fixed to be able to trace memory blocks allocated by PyMem_RawMalloc().
vstinner commentedon Oct 1, 2013
For the development, it would also be nice to fix bpo-18948.
30 remaining items
python-dev commentedon Nov 23, 2013
New changeset 6e2089dbc5ad by Victor Stinner in branch 'default':
Issue bpo-18874: Implement the PEP-454 (tracemalloc)
http://hg.python.org/cpython/rev/6e2089dbc5ad
python-dev commentedon Nov 23, 2013
New changeset 66db0c66a6ee by Victor Stinner in branch 'default':
Issue bpo-18874: Remove tracemalloc.set_traceback_limit()
http://hg.python.org/cpython/rev/66db0c66a6ee
vajrasky commentedon Nov 25, 2013
Here is the patch to tidy up the Lib/test_tracemalloc.py.
It fixed the typo, and removed unused import and unused variables.
vstinner commentedon Nov 25, 2013
Thanks, I applied your patch.
changeset: 87547:841dec769a04
tag: tip
user: Victor Stinner <victor.stinner@gmail.com>
date: Mon Nov 25 09:29:45 2013 +0100
files: Lib/test/test_tracemalloc.py
description:
Cleanup test_tracemalloc.py. Patch written by Vajrasky Kok.
vstinner commentedon Nov 25, 2013
Oh, except:
+ [allocate_bytes(123) for count in range(1000)]
If you don't store the result, the memory is immediatly released.
vstinner commentedon Nov 25, 2013
@neologix: I run test_tracemalloc on "IRIX64 silicon 6.5 07202013 IP35" (our IRIX buildbot) which uses a 32-bit MIPS CPU, and the test pass. The code was compiled with gcc 3.4.6. So packed structures works also on 32-bit MIPS. Well, it is not surprising, int and void* have the same size: 32 bit.
python-dev commentedon Nov 25, 2013
New changeset 2da28004dfac by Victor Stinner in branch 'default':
Issue bpo-18874: tracemalloc: explain the purpose of get_traces.tracebacks in a comment
http://hg.python.org/cpython/rev/2da28004dfac
New changeset b6414aa8cf77 by Victor Stinner in branch 'default':
Issue bpo-18874: apply Jim Jewett's patch on tracemalloc doc
http://hg.python.org/cpython/rev/b6414aa8cf77
python-dev commentedon Nov 25, 2013
New changeset a2811425dbde by Victor Stinner in branch 'default':
Issue bpo-18874: allow to call tracemalloc.Snapshot.statistics(cumulative=True)
http://hg.python.org/cpython/rev/a2811425dbde
python-dev commentedon Nov 26, 2013
New changeset cf2c4cd08dd9 by Victor Stinner in branch 'default':
Issue bpo-18874: tracemalloc: Comment the trace_t structure
http://hg.python.org/cpython/rev/cf2c4cd08dd9
New changeset f06a50c2bf85 by Victor Stinner in branch 'default':
Issue bpo-18874: make it more explicit than set_reentrant() only accept 0 or 1
http://hg.python.org/cpython/rev/f06a50c2bf85
New changeset 02668e24d72d by Victor Stinner in branch 'default':
Issue bpo-18874: Fix typo
http://hg.python.org/cpython/rev/02668e24d72d
vstinner commentedon Nov 26, 2013
The code has been merged. I didn't see any test_tracemalloc on buildbots. I tried to address all remarks on the Rietveld reviews. So I'm now closing the issue.
Please open new issue if you have more remarks. For example, I opened:
Thanks for your reviews.
jimjjewett commentedon Nov 27, 2013
These comments refer to http://hg.python.org/cpython/file/5c9af8194d3b/Doc/library/tracemalloc.rst which is newer than the current patches.
I believe I have consolidated the still-open issues that I see (including those that aren't mine) for this file:
http://hg.python.org/cpython/file/5c9af8194d3b/Doc/library/tracemalloc.rst
(line 24, new comment; also repeated at line 315)
frame (1 frame). To store 25 frames at startup: set the
:envvar:`PYTHONTRACEMALLOC` environment variable to ``25``, or use the
:option:`-X` ``tracemalloc=25`` command line option.
->
frame (1 frame). To store 25 frames at startup: set the
:envvar:`PYTHONTRACEMALLOC` environment variable to ``25``, or use the
:option:`-X` ``tracemalloc=25`` command line option. If tracing is
started later, the maximum number of frames is a parameter to the
:func:`start` function.
(line 111)
If the system has little free memory, snapshots can be written on disk using
the :meth:`Snapshot.dump` method to analyze the snapshot offline. Then use the
:meth:`Snapshot.load` method reload the snapshot.
->
Snapshots can be stored to disk using :meth:`Snapshot.dump` and reloaded using :meth:`Snapshot.load`.
(line 180, new comment)
... The traceback is
where the :mod:`importlib` loaded data most recently: on the ``import pdb``
line of the :mod:`doctest` module. The traceback may change if a new module is
loaded.
->
The traceback represents the call stack when those allocations were made.
As this particular summary is itemized by traceback, it also represents
the call stack for all 903 of those allocations.
(line 252: new comment)
.. function:: clear_traces()
Clear traces of memory blocks allocated by Python.
Add "Future allocations will still be traced."
Is this just a shortcut for stop();start() ? If so, please say so. If not, please explain why.
(Line 311: new comment)
to measure how much memory is used
->
for an approximate measure of how much memory is used
I understand your reluctance to get into too many details, but ... well, it isn't a complete measure. It misses the python objects of tracemalloc itself, it misses the overhead of the module, it misses any filenames that were kept alive only by tracing, etc.
(Line 372, new comment)
If *inclusive* is ``True`` (include), only trace memory blocks allocated
in a file with a name matching :attr:`filename_pattern` at line number
:attr:`lineno`.
->
If *inclusive* is ``True`` (include), then trace memory blocks allocated
in a file with a name matching :attr:`filename_pattern` at line number
:attr:`lineno`. Also excludes any blocks not selected by any filter.
(Line 394: new comment)
This attribute is ignored if the traceback limit is less than
2.->
This attribute is meaningless if the traceback limit is
1.Just after Line 428, not quite a new issue:
Compute the differences with an old snapshot. Get statistics as a sorted
list of :class:`StatisticDiff` instances grouped by *group_by*.
->
(Line 454, new comment)
All inclusive filters are applied at once, a trace is ignored if no
inclusive filters match it. A trace is ignored if at least one exclusive
filter matchs it.
->
If there are any inclusive filters, then they are all applied, and traces which do not match at least one of them are excluded. A trace is also ignored if at least one exclusive
filter matches it.
Line 544, new comment:
.. attribute:: count_diff
->
.. attribute:: count_diff
Line 558, similar to above:
the new snapshots (
int):0if the memory blocks have beenallocated in the new snapshot.
->
the new snapshots (
int). size_diff==size if the memory blocks do not appear in the old snapshot.jimjjewett commentedon Nov 27, 2013
Drat: forgot one at line 277
.. function:: get_traced_memory()
Get the current size and maximum size of memory blocks traced by the
:mod:`tracemalloc` module as a tuple: ``(size: int, max_size: int)``.
I have a tendency to read "maximum size" as the most it is willing/able to trace, rather than the most it has traced at a single time so far. I therefore prefer "peak_size".
Also, this doesn't suggests that the object is strictly a tuple, and that results.size or results.peak_size will not work; is that intentional?
vstinner commentedon Nov 27, 2013
@jim: This issue has been closed, please don't comment closed issues. I created a new issue for your new comments: bpo-19818.