Repository navigation
accept bytes in json.loads() #55185
Description
| BPO | 10976 |
|---|---|
| Nosy | @loewis, @warsaw, @birkenfeld, @ncoghlan, @kousu, @ezio-melotti, @merwok, @bitdancer, @vadmium, @serhiy-storchaka, @jleedev |
| Superseder | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2016-09-10.10:21:47.110>
created_at = <Date 2011-01-21.19:01:47.736>
labels = ['type-feature', 'library']
title = 'accept bytes in json.loads()'
updated_at = <Date 2016-09-10.10:21:47.108>
user = 'https://bugs.python.org/hhas'bugs.python.org fields:
activity = <Date 2016-09-10.10:21:47.108>
actor = 'ncoghlan'
assignee = 'none'
closed = True
closed_date = <Date 2016-09-10.10:21:47.110>
closer = 'ncoghlan'
components = ['Library (Lib)']
creation = <Date 2011-01-21.19:01:47.736>
creator = 'hhas'
dependencies = []
files = ['20481']
hgrepos = []
issue_num = 10976
keywords = ['patch']
message_count = 28.0
messages = ['126772', '126782', '126785', '126786', '126788', '126831', '126986', '126997', '133645', '133672', '145343', '145345', '159359', '159360', '159364', '159366', '159368', '159388', '159391', '159395', '159454', '159469', '204810', '204937', '204959', '215529', '229973', '275615']
nosy_count = 17.0
nosy_names = ['loewis', 'barry', 'georg.brandl', 'ncoghlan', 'hhas', 'kousu', 'ezio.melotti', 'eric.araujo', 'r.david.murray', 'cvrebert', 'docs@python', 'antlong', 'martin.panter', 'serhiy.storchaka', 'Balthazar.Rouberol', 'jleedev', 'Hanxue.Lee']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'needs patch'
status = 'closed'
superseder = '17909'
type = 'enhancement'
url = 'https://bugs.python.org/issue10976'
versions = ['Python 3.6']
Activity
hhas commentedon Jan 21, 2011
json.loads() accepts strings but errors on bytes objects. Documentation and API indicate that both should work. Review of json/init.py code shows that the loads() function's 'encoding' arg is ignored and no decoding takes place before the object is passed to JSONDecoder.decode()
Tested on Python 3.1.2 and Python 3.2rc1; fails on both.
Example:
#################################################
#!/usr/local/bin/python3.2
#################################################
Patch attached.
bitdancer commentedon Jan 21, 2011
Hmm. According to bpo-4136, all bytes support was supposed to have been removed.
pitrou commentedon Jan 21, 2011
Indeed, the documentation (and function docstring) needs fixing instead. It's a pity we didn't remove the useless
encodingparameter.merwok commentedon Jan 21, 2011
Georg: Is it still time to deprecate the encoding parameter in 3.2?
pitrou commentedon Jan 21, 2011
I've committed a doc fix in r88137.
hhas commentedon Jan 22, 2011
Doc fix works for me.
antlong commentedon Jan 25, 2011
Works for me, py2.7 on snow leopard.
bitdancer commentedon Jan 25, 2011
anthony: this is python3-only problem.
ezio-melotti commentedon Apr 13, 2011
Now it's too late for 3.2, should this be done for 3.3?
merwok commentedon Apr 13, 2011
If you’re talking about deprecating the obsolete encoding argument (maybe it’s time for a new bug report), +1.
12 remaining items
[-]json.loads() throws TypeError on bytes object[/-][+]json.loads() raises TypeError on bytes object[/+]serhiy-storchaka commentedon Apr 26, 2012
I mean a string that starts with '\u0000'. b'"\x00...'.
pitrou commentedon Apr 26, 2012
Le jeudi 26 avril 2012 à 15:48 +0000, Serhiy Storchaka a écrit :
According to the RFC, that should be escaped:
All Unicode characters may be placed within the
quotation marks except for the characters that must be escaped:
quotation mark, reverse solidus, and the control characters (U+0000
through U+001F).
And indeed:
serhiy-storchaka commentedon Apr 26, 2012
According to current implementation this is acceptable.
pitrou commentedon Apr 27, 2012
Then perhaps auto-detection can be restricted to strict mode? Non-strict mode would always use utf-8.
Or we can just skip auto-detection altogether (I don't think many people produce utf-16 or utf-32 JSON; that would be a waste of bandwidth for no obvious benefit).
serhiy-storchaka commentedon Apr 27, 2012
Related to this question is a question about errors. How to inform the user, if an error occurred in the decoding with detected encoding? Leave UnicodeDecodeError or convert it to ValueError? If there is a syntax error in JSON -- exception will refer to the position in the decoded string, we should to translate it to the position in the original binary string?
ncoghlan commentedon Nov 30, 2013
bpo-19837 is the complementary problem on the serialisation side - users migrating from Python 2 are accustomed to being able to use the json module directly as a wire protocol module, but the strict Python 3 interpretation as a text transform means that isn't possible - you have to apply the text encoding step separately.
What appears to have happened is that the way JSON is used in practice has diverged from JSON as a formal spec.
Formal spec (this is what the Py3k JSON module implements, and Py2 implements with ensure_ascii=False): JSON is a Unicode text transform, which may optionally be serialised as UTF-8, UTF-16 or UTF-32.
Practice (what the Py2 JSON module implements with ensure_ascii=True, and what is covered in RFC 4627): JSON is a UTF-8 encoded wire protocol
So now we're left with the options:
I'm currently leaning towards the "jsonb" module option, and deprecating the "encoding" argument in the pure text version. It's not pretty, but I think it's better than the alternatives.
loewis commentedon Dec 1, 2013
Bike-shedding: instead of jsonb, make it json.bytes. Else, it may get confused with other protocols, such as "JSONP" or "BSON".
ncoghlan commentedon Dec 1, 2013
json.bytes would also work for me. It wouldn't need to replicate the full
main module API, just combine the text transform with UTF-8 encoding and
decoding (as well as autodetected UTF-16 and UTF-32 decoding) for the main
4 functions (dump[s], load[s]).
If people want UTF-16 and UTF-32 *en*coding (which seem to be rarely used
in combination with JSON), then they can invoke the text transform version
directly, and then do a separate encoding step.
HanxueLee commentedon Apr 4, 2014
This seems to be an issue (bug?) for Python 3.3 When calling json.loads() with a byte array, this is the error
TypeError: can't use a string pattern on a bytes-like object
When I decode the byte array to string
json.loads(response.data.decode(), 'latin-1')
I get this error
TypeError: bytes or integer address expected instead of str instance
vadmium commentedon Oct 25, 2014
bpo-17909 (auto-detecting JSON encoding) looks like it has a patch which would probably satisfy this issue
[-]json.loads() raises TypeError on bytes object[/-][+]accept bytes in json.loads()[/+]ncoghlan commentedon Sep 10, 2016
As Martin noted, Serhiy has implemented the autodetection option for json.loads in bpo-17909 so closing this one as out of date - UTF-8, UTF-16 and UTF-32 encoded JSON data will be deserialised automatically in 3.6, while other text encodings aren't officially supported by the JSON RFCs.