Repository navigation
json fails to serialise numpy.int64 #68501
Description
| BPO | 24313 |
|---|---|
| Nosy | @pitrou, @bitdancer, @njsmith, @eli-b, @serhiy-storchaka, @isidentical, @vlbrown, @mxposed |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2015-05-28.08:32:31.477>
labels = ['type-bug', '3.7']
title = 'json fails to serialise numpy.int64'
updated_at = <Date 2022-02-24.00:38:48.258>
user = 'https://bugs.python.org/thomas-arildsen'bugs.python.org fields:
activity = <Date 2022-02-24.00:38:48.258>
actor = 'mxposed'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2015-05-28.08:32:31.477>
creator = 'thomas-arildsen'
dependencies = []
files = ['39530']
hgrepos = []
issue_num = 24313
keywords = []
message_count = 17.0
messages = ['244288', '244321', '244352', '244355', '244359', '244363', '244370', '244371', '254734', '257451', '257455', '257459', '350567', '350581', '355133', '355143', '413869']
nosy_count = 10.0
nosy_names = ['pitrou', 'r.david.murray', 'njs', 'Eli_B', 'serhiy.storchaka', 'thomas-arildsen', 'Amit Feller', 'BTaskaya', 'vlbrown', 'mxposed']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue24313'
versions = ['Python 3.7']
Activity
thomas-arildsen commentedon May 28, 2015
When I run the attached example in Python 2.7.9, it succeeds. In Python 3.4, it fails as shown below. I use json 2.0.9 and numpy 1.9.2 with both versions of Python. Python and all packages provided by Anaconda 2.2.0.
The error seems to be caused by the serialised object containing a numpy.int64 type. It might fail with other 64-bit numpy types as well (untested).
---------------------------------------------------------------------------
TypeError Traceback (most recent call last) /home/tha/tmp/debug_json/debug_json.py in <module>() 4 test = {'value': np.int64(1)} 5 ----> 6 obj=json.dumps(test)/home/tha/.conda/envs/python3/lib/python3.4/json/init.py in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw)
228 cls is None and indent is None and separators is None and
229 default is None and not sort_keys and not kw):
--> 230 return _default_encoder.encode(obj)
231 if cls is None:
232 cls = JSONEncoder
/home/tha/.conda/envs/python3/lib/python3.4/json/encoder.py in encode(self, o)
190 # exceptions aren't as detailed. The list call should be roughly
191 # equivalent to the PySequence_Fast that ''.join() would do.
--> 192 chunks = self.iterencode(o, _one_shot=True)
193 if not isinstance(chunks, (list, tuple)):
194 chunks = list(chunks)
/home/tha/.conda/envs/python3/lib/python3.4/json/encoder.py in iterencode(self, o, _one_shot)
248 self.key_separator, self.item_separator, self.sort_keys,
249 self.skipkeys, _one_shot)
--> 250 return _iterencode(o, 0)
251
252 def _make_iterencode(markers, _default, _encoder, _indent, _floatstr,
/home/tha/.conda/envs/python3/lib/python3.4/json/encoder.py in default(self, o)
171
172 """
--> 173 raise TypeError(repr(o) + " is not JSON serializable")
174
175 def encode(self, o):
TypeError: 1 is not JSON serializable
bitdancer commentedon May 28, 2015
All python3 ints are what used to be long ints in python2, so the code that recognized short ints no longer exists. Do the numpy types implement __index__? It looks like json doesn't check for __index__, and I wonder if it should.
pitrou commentedon May 28, 2015
I don't know. Simply, under 2.7, int64 inherits from int:
>>> np.int64.__mro__ (<type 'numpy.int64'>, <type 'numpy.signedinteger'>, <type 'numpy.integer'>, <type 'numpy.number'>, <type 'numpy.generic'>, <type 'int'>, <type 'object'>)>>> np.int64.__mro__ (<class 'numpy.int64'>, <class 'numpy.signedinteger'>, <class 'numpy.integer'>, <class 'numpy.number'>, <class 'numpy.generic'>, <class 'object'>)bitdancer commentedon May 29, 2015
Ah, so this is a numpy bug?
serhiy-storchaka commentedon May 29, 2015
Yes, it looks as a bug (or rather lack of feature) in numpy, but numpy have no chance to fix it without help from Python. The json module is not flexible enough.
For now this issue can be workarounded only from user side, with special default handler.
pitrou commentedon May 29, 2015
I wouldn't call it a bug in Numpy (a quirk perhaps?). Numpy ints are fixed-width ints, so some of them can inherit from Python int in 2.x, but not in 3.x.
But not all of them do, since the bitwidth can be different:
bitdancer commentedon May 29, 2015
So in python2, some were json serializable and some weren't? Yes, I'd call that a quirk :)
So back to the question of whether it makes sense for json to look for __index__ to decide if something can be serialized as an int. If not, I don't think there is anything we can do.
pitrou commentedon May 29, 2015
I don't know about __index__, but there's the ages-old discussion of allowing some kind of __json__ hook on types. Of course, none of those solutions would allow round-tripping.
eli-b commentedon Nov 16, 2015
On 64-bit Windows, my 64-bit Python 2.7.9 and my 32-bit 2.7.10 Python both reproduce the failure with a similar traceback.
thomas-arildsen commentedon Jan 4, 2016
Is there any possibility that json could implement special handling of NumPy types? This "lack of a feature" seems to have propagated back into Python 2.7 now in some recent update...
njsmith commentedon Jan 4, 2016
Nothing's changed in python 2.7. Basically: (a) no numpy ints have ever serialized in py3. (b) in py2, either np.int32 *xor* np.int64 will serialize correctly, and which one it is depends on sizeof(long) in the C compiler used to build Python. (This follows from the fact that in py2, the Python 'int' type is always the same size as C 'long'.)
So the end result is: on OS X and Linux, 32-bit Pythons can JSON-serialize np.int32 objects, and 64-bit Pythons can JSON-serialize np.int64 objects, because 64-bit OS X and Linux is ILP64. On Windows, both 32- and 64-bit Pythons can JSON-serialize np.int32 objects, and can't serialize np.int64 objects, because 64-bit Windows is LLP64.
thomas-arildsen commentedon Jan 4, 2016
Thanks for the clarification.
2 remaining items
vlbrown commentedon Aug 26, 2019
Note also that pandas DataFrame.to_json() method has no issue with int64. Perhaps you could borrow their code.
isidentical commentedon Oct 22, 2019
What is the next step of this 4-year-old issue? I think i can prepare a patch for using __index__ (as suggested by @r.david.murray)
serhiy-storchaka commentedon Oct 22, 2019
We could use __index__ for serializing numpy.int64. But what to do with numpy.float32 and numpy.float128? It is a part of a much larger problem (which includes other numbers, collections, encoded strings, named tuples and data classes, etc). I am working on it, but there is a lot of work.
mxposed commentedon Feb 24, 2022
Just ran into this. Are there any updates? Is there any task to contribute to regarding this?
petsuter commentedon May 21, 2022
See #71549
flofriday commentedon Feb 10, 2024
As far as I can see, this issue can be closed.
There is no president in giving some libraries like numpy special treatment in the interpreter (even if they are popular) and a more general discussion for allowing custom hooks for json serialisation already has their own discussion as previously mentioned.