Pickling of NoneType raises PicklingError

@birkenfeld

BPO	6477
Nosy	@birkenfeld, @jaraco, @ncoghlan, @abalkin, @pitrou, @avassalotti, @tifv
Files	backport_pickle_singleton_types_fix.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/avassalotti'
closed_at = <Date 2013-12-01.19:44:48.660>
created_at = <Date 2009-07-13.17:20:39.843>
labels = ['easy', 'type-bug', 'docs']
title = 'Pickling of NoneType raises PicklingError'
updated_at = <Date 2013-12-01.19:44:48.659>
user = 'https://github.com/tifv'

bugs.python.org fields:

activity = <Date 2013-12-01.19:44:48.659>
actor = 'alexandre.vassalotti'
assignee = 'alexandre.vassalotti'
closed = True
closed_date = <Date 2013-12-01.19:44:48.660>
closer = 'alexandre.vassalotti'
components = ['Documentation']
creation = <Date 2009-07-13.17:20:39.843>
creator = 'july'
dependencies = []
files = ['32915']
hgrepos = []
issue_num = 6477
keywords = ['patch', 'easy']
message_count = 17.0
messages = ['90496', '90538', '90540', '90555', '90641', '90648', '108957', '141106', '204862', '204866', '204867', '204869', '204870', '204871', '204874', '204877', '204951']
nosy_count = 9.0
nosy_names = ['georg.brandl', 'jaraco', 'ncoghlan', 'belopolsky', 'pitrou', 'alexandre.vassalotti', 'hagen', 'july', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue6477'
versions = ['Python 2.7']

Python 3.2a0 (py3k:73749M, Jul  1 2009, 23:17:59)
[GCC 4.3.2 [gcc-4_3-branch revision 141291]] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
[40072 refs]
>>> pickle.dumps(type(None))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.2/pickle.py", line 1358, in dumps
    Pickler(f, protocol, fix_imports=fix_imports).dump(obj)
_pickle.PicklingError: Can't pickle <class 'NoneType'>: attribute 
lookup builtins.NoneType failed
[40137 refs]
>>>

I don't see why you want to pickle NoneType. Do you have a proper
use-case for this, or are you just playing around with pickle?

No, my program failed on this. It was not a big problem to manage this,
but I think it is a bug. And it isn't documented (or I can't find it).

Other built-in types have no such problem. Is there something special
with NoneType?

but I think it is a bug

I think it is either a feature request (make NoneType picklable) or a
documentation issue (document that it's not).

Given that dumps(type(Ellipsis)) and dumps(type(NotImplemented)) don't
work either, I am reclassifying this as a documentation bug.

The thing about the types of these three objects (None, Ellipsis,
NotImplemented) is that they are all designed to be singletons and they
all disallow creation of new instances of them.

>>> type(None)()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: cannot create 'NoneType' instances
>>> type(NotImplemented)()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: cannot create 'NotImplementedType' instances
>>> type(Ellipsis)()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: cannot create 'ellipsis' instances

This does mean None has to be special cased if pickling "(object,
type(object))" pairs, but passing types through pickle strikes me as a
dubious enough exercise that it would take a fairly convincing use case
to accept an RFE to make NoneType pickleable (note that any such patch
would have to make sure that doing loads() on such a pickle didn't
manage to create a second copy of NoneType or None).

I agree with Nick.

And if you really want to, you could hack a Pickler subclass to support
NoneType:

import io
import pickle

class XPickler(pickle.Pickler):
  def persistent_id(self, obj):
    if obj is type(None):
      return "NoneType"
    return None

class XUnpickler(pickle.Unpickler):
  def persistent_load(self, persistent_id):
    if persistent_id == "NoneType":
      return type(None)

def dumps(obj):
    f = io.BytesIO()
    XPickler(f).dump(obj)
    return f.getvalue()

def loads(s):
    return XUnpickler(io.BytesIO(s)).load()

Not super elegant, but it works:

>>> loads(dumps([type(None), None]))
[<type 'NoneType'>, None]

Ellipsis and NotImplemented are different from None because they are not pickleable themselves. The None situation is more similar to that of say module level functions:

>>> def f(): pass
... 
>>> dumps(f)  # works
b'\x80\x03c__main__\nf\nq\x00.'
>>> dumps(type(f))  # does not work
Traceback (most recent call last):
  ..
_pickle.PicklingError: Can't pickle <class 'function'>: attribute lookup builtins.function failed

I think documentation at http://docs.python.org/dev/py3k/library/pickle.html#what-can-be-pickled-and-unpickled

Can be improved in two ways:

s/types/objects/ in "The following types can be pickled"
Add a note that "type(x) can be pickled" does not follow from "x can be pickled." The object type(x) must itself fall into one on the listed categories.
"built-in functions defined at the top level of a module" should probably be "built-in classes or functions defined at the top level of the builtins module"
"instances of such classes" should be "instances of picklable classes"
"__setstate__() is picklable" should be "output of __getstate__() is picklable"

I've encountered a use-case where the need to pickle NoneType is more relevant (and difficult to work around).

We have a strongly-type Limited Python language, LimPy, which is based on Python (https://bitbucket.org/yougov/limpy). This parser takes, as part of its initialization arguments, a type specification (indicating which types are allowed and not allowed). In some cases, the return value may be None, in which case the specification says the type must be NoneType.

We're attempting to run this parser in a separate process, using the multiprocessing module, which requires that the arguments passed to and from the parser be pickleable. Unfortunately, because NoneType is in the type specification, it cannot be passed to the parser.

Here's an example of one such type specification (from the test suite):

    class SomeFunctionNamespace:
        @signature([IListType], [], None, ListOfInt)
        def listcount(self, l):
            return range(len(l))

    @signature([IListType], [], None, int)
    def listlen(self, l):
        return len(l)

    # ...

    @signature([IIntType], [IStringType], IStringType, NoneType)
    def givespec(self, *args):
        for a in args:
            print a

Note that we can pickle str and int just fine. Only type(None) fails.

It would be possible to re-write the entire LimPy system (and its child projects) to use a different object where currently NoneType is used, though NoneType is precisely the right thing to be used here except that it can't be pickled.

Since type(None) is a fundamental Python type, it strikes me as a bug that it's not pickleable, though I concede that it's also reasonable to interpret this issue as a feature request (as it's never been pickleable).

Nick makes some good comments that pickling of NoneType should be done right, but other than that, there haven't been any reasons why in principle NoneType should not be pickleable.

NoneType is more akin to str and int which are pickleable than it is to a function or type(NotImplemented), and this is evident by the way that LimPy uses it (before there was any consideration for pickleability).

Therefore, I propose we reconsider that NoneType should be made pickleable.

New changeset 16eba94d3cfe by Alexandre Vassalotti in branch '3.3':
Issue bpo-6477: Added support for pickling the types of built-in singletons.
http://hg.python.org/cpython/rev/16eba94d3cfe

New changeset ff56f48b3277 by Alexandre Vassalotti in branch 'default':
Issue bpo-6477: Merge with 3.3.
http://hg.python.org/cpython/rev/ff56f48b3277

New changeset fbb97f6eb3b3 by Alexandre Vassalotti in branch '2.7':
Issue bpo-6477: Added pickling support for singletons and their types.
http://hg.python.org/cpython/rev/fbb97f6eb3b3

Uh... Your commits make PyNone_Type and PyNotImplemented_Type public APIs, which I don't think is ok, especially not in bugfix releases.

Also, it would be nice to post patches on the tracker before committing (not for trivial patches of course, but it's generally nicer).

Would you be okay with removing the static declaration of PyNotImplemented_Type and PyNone_Type if we prefix their name with an underscore? There isn't any other way to fix this without making the types linkable.

I might revert the 2.7 change anyway as it broke test_xpickle and it doesn't look easy to fix.

New changeset d964d7023aa4 by Alexandre Vassalotti in branch '2.7':
Issue bpo-6477: Revert fbb97f6eb3b3 as it broke test_xpickle.
http://hg.python.org/cpython/rev/d964d7023aa4

Yeah, adding an underscore is a fine way of dealing with this if the symbol has to be exported.

New changeset 9fcba15d7685 by Alexandre Vassalotti in branch '3.3':
Issue bpo-6477: Keep PyNotImplemented_Type and PyNone_Type private.
http://hg.python.org/cpython/rev/9fcba15d7685

New changeset 7d6c27fa7f32 by Alexandre Vassalotti in branch 'default':
Issue bpo-6477: Merge with 3.3.
http://hg.python.org/cpython/rev/7d6c27fa7f32

Antoine, are you okay with applying this fix to 2.7? Or should we just mark this as a won't fix?

I thought it over. I don't think the fix is appropriate for 2.x, as it seems closer to being an extra feature than a bug fix.

Thanks Antoine for calling me out on this one.

Navigation Menu

Repository navigation

Pickling of NoneType raises PicklingError #50726

1 remaining item

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Pickling of NoneType raises PicklingError #50726

Description

Activity

tifv commented on Jul 13, 2009

avassalotti commented on Jul 15, 2009

tifv commented on Jul 15, 2009

hagen commented on Jul 16, 2009

ncoghlan commented on Jul 17, 2009

avassalotti commented on Jul 17, 2009

abalkin commented on Jun 29, 2010

jaraco commented on Jul 25, 2011

1 remaining item

python-dev commented on Dec 1, 2013

python-dev commented on Dec 1, 2013

pitrou commented on Dec 1, 2013

avassalotti commented on Dec 1, 2013

python-dev commented on Dec 1, 2013

pitrou commented on Dec 1, 2013

python-dev commented on Dec 1, 2013

avassalotti commented on Dec 1, 2013

avassalotti commented on Dec 1, 2013

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions