Repository navigation
Cannot override JSON encoding of basic type subclasses #56866
Description
| BPO | 12657 |
|---|---|
| Nosy | @warsaw, @rhettinger, @ezio-melotti, @merwok, @durban, @amirouche, @serhiy-storchaka, @sfreilich |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/serhiy-storchaka'
closed_at = None
created_at = <Date 2011-07-29.18:28:07.772>
labels = ['3.8', 'type-bug', 'library']
title = 'Cannot override JSON encoding of basic type subclasses'
updated_at = <Date 2021-04-01.15:31:03.481>
user = 'https://github.com/warsaw'bugs.python.org fields:
activity = <Date 2021-04-01.15:31:03.481>
actor = 'sfreilich'
assignee = 'serhiy.storchaka'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2011-07-29.18:28:07.772>
creator = 'barry'
dependencies = []
files = []
hgrepos = []
issue_num = 12657
keywords = []
message_count = 9.0
messages = ['141406', '142458', '231983', '231985', '329956', '329958', '329960', '389995', '389996']
nosy_count = 11.0
nosy_names = ['barry', 'rhettinger', 'ezio.melotti', 'eric.araujo', 'daniel.urban', 'Tom.Brown', 'nikow', 'amirouche', 'serhiy.storchaka', 'wiz21', 'sfreilich']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue12657'
versions = ['Python 3.8']Metadata
Metadata
Assignees
Labels
Projects
- Status
Activity
warsaw commentedon Jul 29, 2011
I found this out while experimenting with enum types that inherit from int. The json library provides for extending the encoder to handle non-basic types; in those cases, your class's .default() method is called. However, it is impossible to override how basic types are encoded.
Worse, if you subclass int, you cannot override how instances of your subclass get encoded, because _iterencode() does isinstance() checks. So enum values which subclass int cannot be properly encoded.
I think the isinstance checks should be changed to explicit type equality tests, e.g.
type(o) is int.nikow commentedon Aug 19, 2011
This issue is also a real pain when using namedtuple. In many situations a namedtuple should be serialized as a dict, but there is no reasonable way to get this behavior.
Earlier solutions (e.g., see http://stackoverflow.com/questions/5906831/serializing-a-python-namedtuple-to-json) are now broken in Python 2.7.
wiz21 commentedon Dec 2, 2014
I'm adding a scenario for this problem, a real life one, so it gives a bit more substance.
I use SQLALchemy. I do queries with it which returns KeyedTuples (an SQLALchemy type). KeyedTuples inherits from tuple. KeyedTuples are, in principle, like NamedTuple. I want to transmit the result of my queries over json. Since KeyedTuple inherit from tuple, json can't serialize them. Of course one might say that that problem is outside the scope of json. But if so, it means we have to first convert KeyedTuples to dict and then pass the dict to json. That's alot of copies to do... This problem is rather big because it affects my whole API (close to a hundred of functions)...
I've looked into the code and as noted above, one could replace 'isinstance(x,tuple)' with 'type(x) == tuple'. But without knowledge of how many people use the flexibility introduced by isinstance, it's dangerous. We could also change the nature of default and say that default is called before any type checking in json (so default is not a 'default' anymore). We could also duplicate the default functionnality (so a function called before any type checks and then a default called if all type checks fail). But for these two last cases, I guess the difficulty is : how do we know the pre-type 'default' was applied correctly ?
Patching is not easy because, at least in my case, the C code path is taken => an easy patch (i.e. full python) would force me out of the C path which may be bad for performance (I didn't measure the difference between the 2 paths).
I see this bug is old and not much commented, should we conclude that nobody cares ? That'd a good news since it'd mean a patch wouldn't hurt many people :-)
wiz21 commentedon Dec 2, 2014
Reading bugs a bit, I see this is quite related to :
http://bugs.python.org/issue14886
stF
TomBrown commentedon Nov 15, 2018
I found this work-around useful https://stackoverflow.com/a/32782927
merwok commentedon Nov 15, 2018
A modern solution for this is to define a singledispatch function (with implementations for your custom types) and pass it as the
defaultparameter to the dump functions.(That won’t avoid the custom → dict copy, but that’s how the module works)
serhiy-storchaka commentedon Nov 15, 2018
I'm working on a large patch which will include also the fix for this issue. This is a significant change and it can affect behavior and performance of existing code, so it can be made only in new version.
sfreilich commentedon Apr 1, 2021
Does that work? I thought the default function only got called for non-serializable types.
I'm running into the same issue with some code that would like to do round-trip serialization of a datastructure (which doesn't need 100% generality of supported values but would like to deal with the existing structure of types). Dealing with set is easy, for example:
But you can't do the equivalent thing for tuple, even if you override encode/iterencode. It seems like the JSON module is making performance versus generality tradeoffs, it doesn't make a fresh call to encode/iterencode for every dict key/value for example. Which is fine, but it would be nice if there was a mode that allowed getting custom behavior for every type, taking the performance cost.
sfreilich commentedon Apr 1, 2021
A fully general solution for this might require a separate way to override the behavior for serializing dict keys (since those have to be serialized as strings).