Repository navigation
Redefinition of malloc(3) family of functions at build time #62378
Description
| BPO | 18178 |
|---|---|
| Nosy | @amauryfa, @abalkin, @tiran, @meadori, @koobs |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2013-08-18.13:14:26.237>
created_at = <Date 2013-06-10.08:11:18.228>
labels = ['ctypes', 'type-crash']
title = 'Redefinition of malloc(3) family of functions at build time'
updated_at = <Date 2013-08-18.13:14:26.235>
user = 'https://bugs.python.org/cvs-src'bugs.python.org fields:
activity = <Date 2013-08-18.13:14:26.235>
actor = 'christian.heimes'
assignee = 'none'
closed = True
closed_date = <Date 2013-08-18.13:14:26.237>
closer = 'christian.heimes'
components = ['ctypes']
creation = <Date 2013-06-10.08:11:18.228>
creator = 'cvs-src'
dependencies = []
files = ['30522']
hgrepos = []
issue_num = 18178
keywords = []
message_count = 7.0
messages = ['190892', '190897', '190926', '192068', '195462', '195493', '195557']
nosy_count = 7.0
nosy_names = ['amaury.forgeotdarc', 'belopolsky', 'christian.heimes', 'meador.inge', 'python-dev', 'koobs', 'cvs-src']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue18178'
versions = ['Python 2.7', 'Python 3.3', 'Python 3.4']
Activity
cvs-src commentedon Jun 10, 2013
Hello.
Marcel Moolenaar (marcel@FreeBSD.org) pointed this out after committing FreeBSD revision 250991 [1], that makes the malloc(3) family of functions weak symbols. I'm citing him, because (silly me) I don't understand all of this completely:
"""
After my commit to head that makes the malloc(3) family of functions weak symbols, a bug in what appears to be the Python build was exposed. The net effect of the bug is that _ctypes.so contains (strong) definitions of the malloc(3) family of functions and resulting in unintended symbol resolution.
_ctypes.so incorporates the libffi functionality for what I presume is
the basis for Python bindings. libffi includes dlmalloc.c, an open source allocator. dlmalloc.c is incuded by closures.c and closures.c defines USE_DL_PREFIX. On top of that closures.c makes all allocator functions static. This, therefore adds a memory allocator to libffi without exposure of standard symbols In short: dlmalloc.c never gets compiler separately or independently for this reason.
The python build however compiles dlmalloc.c separately/independently. As such, dlmalloc.c now defines and exports the malloc(3) family of functions and it also get linked into _ctypes.so. Thus when libffi is built as part of the Python build, the set of symbols exported is different from when libffi is compiled as a port.
The simplest test case is this (on a -current machines):
The build of databases/py-sqlite3 fails because this:
running config
*** Signal 11
On amd64 I observed an assertion failure of FreeBSD's malloc, with
the same effect.
Manually triggering this:
fbsdvm% pwd
/usr/ports/databases/py-sqlite3/work/Python-2.7.5/Modules
fbsdvm% /usr/local/bin/python2.7 setup.py config
running config
Segmentation fault
But if symbols are resolved non-lazily, things change:
fbsdvm% setenv LD_BIND_NOW yes
fbsdvm% /usr/local/bin/python2.7 setup.py config
running config
This demonstrates how, after loading _ctypes.so, malloc(3) and friends
get resolved differently and thus inconsistently from before.
The publicly visible symbols in _ctypes.so, also show the problem:
fbsdvm% nm /usr/local/lib/python2.7/lib-dynload/_ctypes.so | grep ' T ' | egrep '(alloc)|(free)'
0000c3f0 T _ctypes_alloc_callback
00005250 T _ctypes_alloc_format_string
00016d10 T calloc
000121e0 T ffi_closure_alloc
00013760 T ffi_closure_free
00016050 T free
000170c0 T independent_calloc
000172d0 T independent_comalloc
000148e0 T malloc
00017460 T malloc_footprint
00017480 T malloc_max_footprint
000175c0 T malloc_stats
00017440 T malloc_trim
000176e0 T malloc_usable_size
000173a0 T pvalloc
00016d90 T realloc
00017310 T valloc
There are definitions of malloc(3) and friends that shouldn't be there.
"""
We have similar reports in our bug-tracker [2] and [3]. And the patch attached fixes this for post-r250991 versions of FreeBSD, and noop for earlier versions. The same patch applied to python2.7, python3.2, python3.3 and patched in FreeBSD ports tree locally.
[1] http://svnweb.freebsd.org/changeset/base/250991
[2] http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/179102
[3] http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/179413
amauryfa commentedon Jun 10, 2013
Said simpler: dlmalloc.c code is indeed compiled twice:
The second one is not necessary, of course. The patch looks good to me.
abalkin commentedon Jun 10, 2013
I don't have a system affected by the change, but the explanation and the patch look right to me.
FWIW, the patch applies cleanly to 3.4 head and passes 'make test'.
koobs commentedon Jun 30, 2013
I've added a new FreeBSD 10.0-CURRENT buildbot to the pool (thanks antoine) that reproduces the issue and should provide sufficient coverage for testing the proposed patch:
http://buildbot.python.org/all/buildslaves/koobs-freebsd10
I'll upgrade the FreeBSD 9-STABLE buildbot once this is resolved
python-dev commentedon Aug 17, 2013
New changeset f09ca52747a6 by Christian Heimes in branch '3.3':
Issue bpo-18178: Fix ctypes on BSD. dlmalloc.c was compiled twice which broke malloc weak symbols.
http://hg.python.org/cpython/rev/f09ca52747a6
New changeset bea2f12e899e by Christian Heimes in branch 'default':
Issue bpo-18178: Fix ctypes on BSD. dlmalloc.c was compiled twice which broke malloc weak symbols.
http://hg.python.org/cpython/rev/bea2f12e899e
New changeset d4ac6eee7061 by Christian Heimes in branch '2.7':
Issue bpo-18178: Fix ctypes on BSD. dlmalloc.c was compiled twice which broke malloc weak symbols.
http://hg.python.org/cpython/rev/d4ac6eee7061
koobs commentedon Aug 17, 2013
Commit looks good, confirming test suite passing for 3.x, 3.3 and 2.7.on http://buildbot.python.org/all/buildslaves/koobs-freebsd10
Thank you for picking this up and finishing it off Christian.
tiran commentedon Aug 18, 2013
Thx for the fix!