Repository navigation
String allocations waste 3 bytes of memory on average. #48695
Description
| BPO | 4445 |
|---|---|
| Nosy | @rhettinger, @mdickinson, @tiran |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/mdickinson'
closed_at = <Date 2008-12-06.15:34:04.137>
created_at = <Date 2008-11-27.12:14:44.506>
labels = ['interpreter-core', 'performance']
title = 'String allocations waste 3 bytes of memory on average.'
updated_at = <Date 2008-12-06.15:34:04.135>
user = 'https://github.com/mdickinson'bugs.python.org fields:
activity = <Date 2008-12-06.15:34:04.135>
actor = 'mark.dickinson'
assignee = 'mark.dickinson'
closed = True
closed_date = <Date 2008-12-06.15:34:04.137>
closer = 'mark.dickinson'
components = ['Interpreter Core']
creation = <Date 2008-11-27.12:14:44.506>
creator = 'mark.dickinson'
dependencies = []
files = ['12149']
hgrepos = []
issue_num = 4445
keywords = ['patch']
message_count = 12.0
messages = ['76495', '76498', '76503', '76512', '76552', '76553', '76554', '76555', '76558', '76577', '77078', '77137']
nosy_count = 3.0
nosy_names = ['rhettinger', 'mark.dickinson', 'christian.heimes']
pr_nums = []
priority = 'normal'
resolution = 'accepted'
stage = None
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue4445'
versions = ['Python 2.6', 'Python 3.0', 'Python 2.7']
Activity
mdickinson commentedon Nov 27, 2008
There are a number of places in Objects/stringobject.c where memory is
allocated for a string of length n using:
PyObject_MALLOC(sizeof(PyStringObject) + n)
On my computer (OS X 10.5.5/Intel), and, I suspect, on most common
platforms, the PyStringObject struct is going to contain some number of
bytes (probably 3) of trailing padding; the result is that the
PyObject_MALLOC call above asks for 3 more bytes than are necessary, and
on average the Python interpreter will waste 3 bytes of memory per string
allocation. Is there any reason not to replace these calls with:
PyObject_MALLOC(offsetof(PyStringObject, ob_sval) + n + 1)
instead?
Patch attached.
mdickinson commentedon Nov 27, 2008
Updated patch: fix overflow checks to use offsetof instead of sizeof as
well.
rhettinger commentedon Nov 27, 2008
+1 on the idea. -1 on the current patch. Can you encapsulate this in a
simpler macro? I find the proposed replacement to be cryptic and
error-prone (i.e. hard to mentally verify its correctness and somewhat
likely to be screwed-up by a future maintainer).
rhettinger commentedon Nov 27, 2008
Suggestion:
mdickinson commentedon Nov 28, 2008
Yep. That works nicely.
Here's a revised patch.
tiran commentedon Nov 28, 2008
Why is +1 required here? If I understand offsetof() correctly than it
returns the position of the ob_sval element. Shouldn't PyStringObject +
offsetof(PyStringObject, ob_sval) point to the first element of the
ob_svall array?
mdickinson commentedon Nov 28, 2008
The +1 is there for the trailing null byte on the string: if s is a
Python string with len(s) == n, then the ob_sval array needs space for n+1
characters.
tiran commentedon Nov 28, 2008
Ah! I forgot the trailing \0 byte ... Thanks Mark!
mdickinson commentedon Nov 28, 2008
Hmmm. test_sys fails on 64-bit build.
Patch updated to fix this.
All tests now pass on 32-bit and 64-bit, debug and non-debug builds.
rhettinger commentedon Nov 29, 2008
Looks much cleaner.
mdickinson commentedon Dec 5, 2008
Applied to the trunk in r67601. Will merge to other branches if the
buildbots look okay.
mdickinson commentedon Dec 6, 2008
Applied to py3k in r67610.