Add *Calloc functions to CPython memory allocation API

@pitrou

BPO	21233
Nosy	@pitrou, @vstinner, @njsmith, @skrah, @MojoVampire
Files	calloc.patch calloc-2.patch calloc-3.patch bench_alloc.py test.c calloc-4.patch use_calloc.patch bench_alloc2.py calloc-5.patch calloc-6.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-06-02.20:29:45.268>
created_at = <Date 2014-04-15.08:56:01.630>
labels = ['interpreter-core', 'type-feature']
title = 'Add *Calloc functions to CPython memory allocation API'
updated_at = <Date 2014-06-02.20:29:45.267>
user = 'https://github.com/njsmith'

bugs.python.org fields:

activity = <Date 2014-06-02.20:29:45.267>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2014-06-02.20:29:45.268>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2014-04-15.08:56:01.630>
creator = 'njs'
dependencies = []
files = ['34897', '34903', '34924', '35052', '35059', '35063', '35064', '35065', '35072', '35097']
hgrepos = []
issue_num = 21233
keywords = ['patch']
message_count = 95.0
messages = ['216281', '216390', '216394', '216399', '216403', '216404', '216422', '216425', '216431', '216433', '216444', '216451', '216452', '216455', '216515', '216567', '216668', '216671', '216681', '216682', '216686', '217228', '217242', '217246', '217251', '217252', '217253', '217254', '217255', '217256', '217257', '217262', '217274', '217276', '217282', '217283', '217284', '217291', '217294', '217295', '217297', '217298', '217302', '217303', '217304', '217305', '217306', '217307', '217308', '217309', '217310', '217323', '217324', '217325', '217326', '217330', '217331', '217333', '217346', '217348', '217349', '217351', '217357', '217375', '217380', '217382', '217423', '217436', '217445', '217549', '217553', '217594', '217617', '217619', '217785', '217786', '217794', '217797', '217826', '217829', '217831', '217832', '217838', '217839', '217840', '217841', '217844', '217866', '217972', '219627', '219628', '219630', '219631', '219634', '219635']
nosy_count = 8.0
nosy_names = ['pitrou', 'vstinner', 'njs', 'skrah', 'neologix', 'python-dev', 'jtaylor', 'josh.r']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue21233'
versions = ['Python 3.5']

Numpy would like to switch to using the CPython allocator interface in order to take advantage of the new tracemalloc infrastructure in 3.4. But, numpy relies on the availability of calloc(), and the CPython allocator API does not expose calloc().
https://docs.python.org/3.5/c-api/memory.html#c.PyMemAllocator

So, we should add *Calloc variants. This met general approval on python-dev. Thread here:
https://mail.python.org/pipermail/python-dev/2014-April/133985.html

This would involve adding a new .calloc field to the PyMemAllocator struct, exposed through new API functions PyMem_RawCalloc, PyMem_Calloc, PyObject_Calloc. [It's not clear that all 3 would really be used, but since we have only one PyMemAllocator struct that they all share, it'd be hard to add support to only one or two of these domains and not the rest. And the higher-level calloc variants might well be used. Numpy array buffers are often small (e.g., holding only a single value), and these small buffers benefit from small-alloc optimizations; meanwhile, large buffers benefit from calloc optimizations. So it might be optimal to use a single allocator that has both.]

We might also have to rename the PyMemAllocator struct to ensure that compiling old code with the new headers doesn't silently leave garbage in the .calloc field and lead to crashes.

Here is a first patch adding the following functions:

  void* PyMem_RawCalloc(size_t n);
  void* PyMem_Calloc(size_t n);
  void* PyObject_Calloc(size_t n);
  PyObject* _PyObject_GC_Calloc(size_t);

It adds the following field after malloc field to PyMemAllocator structure:

  void* (*calloc) (void *ctx, size_t size);

It changes the tracemalloc module to trace "calloc" allocations, add new tests and document new functions.

The patch also contains an important change: PyType_GenericAlloc() uses calloc instead of malloc+memset(0). It may be faster, I didn't check.

So what is the point of _PyObject_GC_Calloc ?

General comment on patch: For the flag value that toggles zero-ing, perhaps use a different name, e.g. setzero, clearmem, initzero or somesuch instead of calloc? calloc already gets used to refer to both the C standard function and the function pointer structure member; it's mildly confusing to have it *also* refer to a boolean flag as well.

Additional comment on clarity: Might it make sense to make the calloc structure member take both the num and size arguments that the underlying calloc takes? That is, instead of:

void* (*calloc) (void *ctx, size_t size);

Declare it as:

void* (*calloc) (void *ctx, size_t num, size_t size);

Beyond potentially allowing more detailed tracing info at some later point (and much like the original calloc, potentially allowing us to verify that the components do not overflow on multiply, instead of assuming every caller must multiply and check for themselves), it also seems like it's a bit more friendly to have the prototype for the structure calloc to follow the same pattern as the other members for consistency (Principle of Least Surprise): A context pointer, plus the arguments expected by the equivalent C function.

Sorry for breaking it up, but the same comment on consistent prototypes mirroring the C standard lib calloc would apply to all the API functions as well, e.g. PyMem_RawCalloc, PyMem_Calloc, PyObject_Calloc and _PyObject_GC_Calloc, not just the structure function pointer.

So what is the point of _PyObject_GC_Calloc ?

It calls calloc(size) instead of malloc(size), calloc() which can be faster than malloc()+memset(), see:
https://mail.python.org/pipermail/python-dev/2014-April/133985.html

_PyObject_GC_Calloc() is used by PyType_GenericAlloc(). If I understand directly, it is the default allocator to allocate Python objects.

In numpy, I found the two following functions:

/*NUMPY_API
 * Allocates memory for array data.
 */
void* PyDataMem_NEW(size_t size);

/*NUMPY_API
 * Allocates zeroed memory for array data.
 */
void* PyDataMem_NEW_ZEROED(size_t size, size_t elsize);

So it looks like it needs two size_t parameters. Prototype of the C function calloc():

void *calloc(size_t nmemb, size_t size);

I agree that it's better to provide the same prototype than calloc().

New patch:

replace "size_t size" with "size_t nelem, size_t elsize" in the prototype of calloc functions (the parameter names come from the POSIX standard)
replace "int calloc" with "int zero" in helper functions

Le 16/04/2014 04:40, STINNER Victor a écrit :

STINNER Victor added the comment:

> So what is the point of _PyObject_GC_Calloc ?

It calls calloc(size) instead of malloc(size)

No, the question is why you didn't simply change _PyObject_GC_Malloc
(which is a private function).

> So what is the point of _PyObject_GC_Calloc ?

It calls calloc(size) instead of malloc(size), calloc() which can be faster than malloc()+memset(), see:
https://mail.python.org/pipermail/python-dev/2014-April/133985.html

It will only make a difference if the allocated region is large enough
to be allocated by mmap (so not for 90% of objects).

>> So what is the point of _PyObject_GC_Calloc ?
>
> It calls calloc(size) instead of malloc(size)

No, the question is why you didn't simply change _PyObject_GC_Malloc
(which is a private function).

Oh ok, I didn't understand. I don't like changing the behaviour of
functions, but it's maybe fine if the function is private.

2014-04-16 3:18 GMT-04:00 Charles-François Natali <report@bugs.python.org>:

> It calls calloc(size) instead of malloc(size), calloc() which can be faster than malloc()+memset(), see:
> https://mail.python.org/pipermail/python-dev/2014-April/133985.html

It will only make a difference if the allocated region is large enough
to be allocated by mmap (so not for 90% of objects).

Even if there are only 10% of cases where it may be faster, I think
that it's interesting to use calloc() to allocate Python objects. You
may create large Python objects ;-)

I didn't check which objects use (indirectly) _PyObject_GC_Calloc().

We need some kind of prominent documentation that existing
programs need to be changed:

My final commit includes an addition to What's New in Python 3.5 doc,
including a notice in the porting section. It is not enough?

Even if the API is public, the PyMemAllocator thing is low level. It's not
part of the stable ABI. Except failmalloc, I don't know any user. I don't
expect a lot of complain and it's easy to port the code.

If WITH_VALGRIND is defined, nbytes is uninitialized in
_PyObject_Alloc().

Did you see my second commit? It's nlt already fixed?

> 5) If WITH_VALGRIND is defined, nbytes is uninitialized in
_PyObject_Alloc().

Did you see my second commit? It's nlt already fixed?

I don't think so, I have revision 5d076506b3f5 here.

"allocate nbytes elements of size 1"

PyObject_Malloc(100) asks to allocate one object of 100 bytes.

For PyMem_Malloc() and PyMem_RawMalloc(), it's more difficult to guess, but
IMO it's sane to bet that a single memory block of size bytes is requested.

I consider that char data[100] is a object of 100 bytes, but you call it
100 object of 1 byte.

I don't think that using nelem or elsize matters in practice.

STINNER Victor <report@bugs.python.org> wrote:

PyObject_Malloc(100) asks to allocate one object of 100 bytes.

Okay, then let's please call it:

_PyObject_Calloc(void *ctx, size_t nobjs, size_t objsize)

_PyObject_Alloc(int use_calloc, void *ctx, size_t nobjs, size_t objsize)

STINNER Victor <report@bugs.python.org> wrote:

My final commit includes an addition to What's New in Python 3.5 doc,
including a notice in the porting section. It is not enough?

I'm not sure: The usual case with ABI changes is that extensions may segfault
if they are *not* recompiled [1]. In that case documenting it in What's New is
standard procedure.

Here the extension *is* recompiled and still segfaults.

Even if the API is public, the PyMemAllocator thing is low level. It's not
part of the stable ABI. Except failmalloc, I don't know any user. I don't
expect a lot of complain and it's easy to port the code.

Perhaps it's worth asking on python-dev. Nathaniel's suggestion isn't bad
either (e.g. name it PyMemAllocatorEx).

[1] I was told on python-dev that many people in fact do not recompile.

New changeset 358a12f4d4bc by Victor Stinner in branch 'default':
Issue bpo-21233: Fix _PyObject_Alloc() when compiled with WITH_VALGRIND defined
http://hg.python.org/cpython/rev/358a12f4d4bc

New changeset 6374c2d957a9 by Victor Stinner in branch 'default':
Issue bpo-21233: Rename the C structure "PyMemAllocator" to "PyMemAllocatorEx" to
http://hg.python.org/cpython/rev/6374c2d957a9

I'm not sure: The usual case with ABI changes is that extensions may segfault if they are *not* recompiled [1].

Ok, I renamed the structure PyMemAllocator to PyMemAllocatorEx, so the compilation fails because PyMemAllocator name is not defined. Modules compiled for Python 3.4 will crash on Python 3.5 if they are not recompiled, but I hope that you recompile your modules when you don't use the stable ABI.

Using PyMemAllocator is now more complex because it depends on the Python version. See for example the patch for pyfailmalloc:
https://bitbucket.org/haypo/pyfailmalloc/commits/9db92f423ac5f060d6ff499ee4bb74ebc0cf4761

Using the C preprocessor, it's possible to limit the changes.

"Okay, then let's please call it:
_PyObject_Calloc(void *ctx, size_t nobjs, size_t objsize)
_PyObject_Alloc(int use_calloc, void *ctx, size_t nobjs, size_t objsize)"

"void * PyMem_RawCalloc(size_t nelem, size_t elsize);" prototype comes from the POSIX standad:
http://pubs.opengroup.org/onlinepubs/009695399/functions/calloc.html

I'm don't want to change the prototype in Python. Extract of Python documentation:

.. c:function:: void* PyMem_RawCalloc(size_t nelem, size_t elsize)

Allocates *nelem* elements each whose size in bytes is *elsize* (...)

New changeset dff6b4b61cac by Victor Stinner in branch 'default':
Issue bpo-21233: Revert bytearray(int) optimization using calloc()
http://hg.python.org/cpython/rev/dff6b4b61cac

"2) I'm not happy with the refactoring in bytearray_init(). (...)

Somewhat similarly, I wonder if it was necessary to refactor
PyBytes_FromStringAndSize(). (...)"

Ok, I reverted the change on bytearray(int) and opened the issue bpo-21644 to discuss these two optimizations.

I reread the issue. I hope that I now addressed all issues. The remaining issue, bytearray(int) is now tracked by the new issue bpo-21644.

Navigation Menu

Repository navigation

Add *Calloc functions to CPython memory allocation API #65432

69 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Add *Calloc functions to CPython memory allocation API #65432

Description

Activity

njsmith commented on Apr 15, 2014

vstinner commented on Apr 15, 2014

pitrou commented on Apr 15, 2014

MojoVampire commented on Apr 15, 2014

MojoVampire commented on Apr 15, 2014

MojoVampire commented on Apr 15, 2014

vstinner commented on Apr 16, 2014

vstinner commented on Apr 16, 2014

vstinner commented on Apr 16, 2014

pitrou commented on Apr 16, 2014

neologix commented on Apr 16, 2014

vstinner commented on Apr 16, 2014

vstinner commented on Apr 16, 2014

69 remaining items

vstinner commented on May 3, 2014

vstinner commented on May 3, 2014

skrah commented on May 3, 2014

vstinner commented on May 3, 2014

skrah commented on May 3, 2014

skrah commented on May 4, 2014

python-dev commented on May 6, 2014

python-dev commented on Jun 2, 2014

vstinner commented on Jun 2, 2014

vstinner commented on Jun 2, 2014

python-dev commented on Jun 2, 2014

vstinner commented on Jun 2, 2014

vstinner commented on Jun 2, 2014

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions