9th import of module _pickle always crashes

@loewis

BPO	11321
Nosy	@loewis, @brettcannon, @amauryfa, @ncoghlan, @pitrou, @vstinner, @avassalotti, @Trundle
Files	issue11321.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2011-07-15.19:04:09.432>
created_at = <Date 2011-02-25.15:51:21.040>
labels = ['interpreter-core', 'type-crash']
title = '9th import of module _pickle always crashes'
updated_at = <Date 2011-07-15.19:04:09.430>
user = 'https://bugs.python.org/palmkevin'

bugs.python.org fields:

activity = <Date 2011-07-15.19:04:09.430>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2011-07-15.19:04:09.432>
closer = 'pitrou'
components = ['Interpreter Core']
creation = <Date 2011-02-25.15:51:21.040>
creator = 'palm.kevin'
dependencies = []
files = ['20931']
hgrepos = []
issue_num = 11321
keywords = ['patch']
message_count = 11.0
messages = ['129384', '129622', '129623', '129624', '129626', '129670', '129674', '129675', '129678', '140458', '140459']
nosy_count = 10.0
nosy_names = ['loewis', 'brett.cannon', 'amaury.forgeotdarc', 'ncoghlan', 'pitrou', 'vstinner', 'alexandre.vassalotti', 'Trundle', 'palm.kevin', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue11321'
versions = ['Python 3.2', 'Python 3.3']

Please have a look to the following app:

#include "Python.h"

main(int argc, char **argv)
{
  int i;
  printf("START\n");
  for(i = 0; i < 20; i++){
    Py_Initialize();

    printf("Try import #%i ...", i);
    PyRun_SimpleString("import _pickle\n");

    printf("SUCCESS\n");
    Py_Finalize();
  }
  printf("END\n");
}

If I run that app (Linux), then I get the following output:

START
Try import #0 ...SUCCESS
Try import #1 ...SUCCESS
Try import #2 ...SUCCESS
Try import #3 ...SUCCESS
Try import #4 ...SUCCESS
Try import #5 ...SUCCESS
Try import #6 ...SUCCESS
Try import #7 ...SUCCESS
Segmentation fault

It systematically crashes on the 9th call ...
Does anybody know the reason of this segmentation fault problem??

I can definitely confirm that one:

~/devel/py3k$ gcc -I/usr/local/include/python3.3m -o bpo-11321 bpo-11321.c -L/usr/local/lib/python3.3 -lpython3.3m -lm -lpthread -ldl -lutil -rdynamic
~/devel/py3k$ ./issue11321
START
Try import #0 ...SUCCESS
Try import #1 ...SUCCESS
Try import #2 ...SUCCESS
Try import #3 ...SUCCESS
Try import #4 ...SUCCESS
Try import #5 ...SUCCESS
Try import #6 ...SUCCESS
Try import #7 ...SUCCESS
Segmentation fault

I can also confirm that it isn't a regression w.r.t 3.1:

~/devel/py3k$ gcc -I/usr/include/python3.1 -o bpo-11321 bpo-11321.c -L/usr/lib/python3.1 -lpython3.1 -lm -lpthread -ldl -lutil -rdynamic
~/devel/py3k$ ./issue11321
START
Try import #0 ...SUCCESS
Try import #1 ...SUCCESS
Try import #2 ...SUCCESS
Try import #3 ...SUCCESS
Try import #4 ...SUCCESS
Try import #5 ...SUCCESS
Try import #6 ...SUCCESS
Try import #7 ...SUCCESS
Segmentation fault

It *is*, however, a regression from the 2.x line (where, presumably, cPickle simply never got reinitialised)

I haven't worked out where the error is happening yet, but the 13 allocated static variables and the complete lack of a module cleanup function aren't going to be helping matters.

Here's a sample stack trace of it blowing up:

Program received signal SIGSEGV, Segmentation fault.
0x0000000000479404 in type_dealloc (type=0x7ffff6a82f20) at Objects/typeobject.c:2523
2523 _PyObject_GC_UNTRACK(type);

(gdb) bt
#0 0x0000000000479404 in type_dealloc (type=0x7ffff6a82f20) at Objects/typeobject.c:2523
#1 0x0000000000465a2b in dict_dealloc (mp=0x8f61b0) at Objects/dictobject.c:1020
#2 0x00000000004e43c8 in _PyImport_FixupExtensionUnicode (mod=0x7ffff7f68ef0,
name=0x7fffffffcd40 "_pickle", filename=0x7ffff7f12570) at Python/import.c:611
#3 0x00000000004e7fcf in _PyImport_LoadDynamicModule (name=0x7fffffffcd40 "_pickle",
pathname=0x7fffffffbc70 "/usr/local/lib/python3.3/lib-dynload/pickle.cpython-33m.so",
fp=<value optimised out>) at ./Python/importdl.c:85
#4 0x00000000004e5f79 in import_submodule (mod=0x796b20, subname=0x7fffffffcd40 "_pickle",
fullname=0x7fffffffcd40 "_pickle") at Python/import.c:2894
#5 0x00000000004e61ef in load_next (mod=0x796b20, altmod=<value optimised out>,
p_name=<value optimised out>, buf=0x7fffffffcd40 "_pickle", p_buflen=0x7fffffffcd38)
at Python/import.c:2706
#6 0x00000000004e6931 in import_module_level (name=0x0, globals=0x922060,
locals=<value optimised out>, fromlist=0x796b20, level=<value optimised out>)
at Python/import.c:2422
#7 0x00000000004e6e74 in PyImport_ImportModuleLevel (name=0x7ffff7e6c550 "_pickle",
globals=0x922060, locals=0x922060, fromlist=0x796b20, level=<value optimised out>)
at Python/import.c:2474
#8 0x00000000004c642b in builtin___import_ (self=<value optimised out>,
args=<value optimised out>, kwds=<value optimised out>) at Python/bltinmodule.c:167
#9 0x000000000042c897 in PyObject_Call (func=0x7ffff7edf8c0, arg=0x8e79e0, kw=0xa2e79)
at Objects/abstract.c:2149
#10 0x00000000004c6c73 in PyEval_CallObjectWithKeywords (func=0x7ffff7edf8c0,
arg=0x7ffff7e85ef0, kw=0xa2e79) at Python/ceval.c:3739
#11 0x00000000004ca6de in PyEval_EvalFrameEx (f=0xae4490, throwflag=<value optimised out>)
at Python/ceval.c:2316
#12 0x00000000004cf343 in PyEval_EvalCodeEx (_co=0x7ffff7ea5690, globals=<value optimised out>,
locals=<value optimised out>, args=0x0, argcount=<value optimised out>,
kws=<value optimised out>, kwcount=0, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0)
at Python/ceval.c:3295
#13 0x00000000004cf5db in PyEval_EvalCode (co=0x7ffff6a82f20, globals=0x8e79e0, locals=0xa2e79)
at Python/ceval.c:761
#14 0x0000000000419c7e in run_mod (command=<value optimised out>, flags=<value optimised out>)
at Python/pythonrun.c:1769
#15 PyRun_StringFlags (command=<value optimised out>, flags=<value optimised out>)
at Python/pythonrun.c:1703
#16 PyRun_SimpleStringFlags (command=<value optimised out>, flags=<value optimised out>)
at Python/pythonrun.c:1276
#17 0x0000000000415c41 in main ()

So _pickle itself doesn't seem to be dying, but it is eventually leaving things in a bad state so that the *next* time around, the import machinery blows up. Note that switching to a different module (e.g. "_struct") eliminates the segfault.

It's because the _pickle module doesn't incref Pickler_Type and Unpickler_Type before calling PyModule_AddObject() (which steals a reference).

The attached patch fixes the issue.

The patch looks good. It may also fix bpo-9197, which fails in a similar fashion

While Andreas's patch does indeed prevent the crash, there is something more going on here.

I modified his patch to print out the reference counts immediately after the new INCREF commands.

With the INCREF commands commented out, it looks like this:

~/devel/py3k$ ./issue113213.3
START
Try import #0 ...Initialising _pickle
Pickler type references: 10
Unpickler type references: 8
Module initialised
SUCCESS
Try import #1 ...Initialising _pickle
Pickler type references: 10
Unpickler type references: 8
Module initialised
SUCCESS
Try import #2 ...Initialising _pickle
Pickler type references: 9
Unpickler type references: 7
Module initialised
SUCCESS
Try import #3 ...Initialising _pickle
Pickler type references: 8
Unpickler type references: 6
Module initialised
SUCCESS
Try import #4 ...Initialising _pickle
Pickler type references: 7
Unpickler type references: 5
Module initialised
SUCCESS
Try import #5 ...Initialising _pickle
Pickler type references: 6
Unpickler type references: 4
Module initialised
SUCCESS
Try import #6 ...Initialising _pickle
Pickler type references: 5
Unpickler type references: 3
Module initialised
SUCCESS
Try import #7 ...Initialising _pickle
Pickler type references: 4
Unpickler type references: 2
Module initialised
SUCCESS
Try import #8 ...Initialising _pickle
Pickler type references: 3
Unpickler type references: 1
Module initialised
Segmentation fault

Note that it does the right thing the first time Py_Finalize is called, but the reference counts start going down after that.

When the INCREF is included, the count goes up initially and then levels out.

So I believe the simple patch is just masking the problem instead of fixing it. The correct answer is to follow the documentation and implement the module finalisation protocol for _pickle (i.e. move all the static globals into the module struct and implement module traversal and finalisation functions).

That initial increase you can see happens because the module's dict is copied in _PyImport_FixupExtensionUnicode().

Looking at the example of _struct.c, I'll withdraw my objection to the patch. The segfault is due to the attempt to destroy a statically allocated type object, and the only viable solution to that is to ensure the reference count never drops to 0, even during Py_Finalize.

I also created bpo-11349 to cover the fact that _pickle really shouldn't be stashing things in static globals now that a better alternative is available.

New changeset 1ae0b7b8de0b by Antoine Pitrou in branch '3.2':
Issue bpo-11321: Fix a crash with multiple imports of the _pickle module when
http://hg.python.org/cpython/rev/1ae0b7b8de0b

New changeset 6674272754da by Antoine Pitrou in branch 'default':
Issue bpo-11321: Fix a crash with multiple imports of the _pickle module when
http://hg.python.org/cpython/rev/6674272754da

Should be fixed now, thank you!

Navigation Menu

Repository navigation

9th import of module _pickle always crashes #55530

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

9th import of module _pickle always crashes #55530

Description

Activity

palmkevin commented on Feb 25, 2011

ncoghlan commented on Feb 27, 2011

ncoghlan commented on Feb 27, 2011

ncoghlan commented on Feb 27, 2011

Trundle commented on Feb 27, 2011

amauryfa commented on Feb 27, 2011

ncoghlan commented on Feb 28, 2011

Trundle commented on Feb 28, 2011

ncoghlan commented on Feb 28, 2011

python-dev commented on Jul 15, 2011

pitrou commented on Jul 15, 2011

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions