Repository navigation
Autodetecting JSON encoding #62109
Description
| BPO | 17909 |
|---|---|
| Nosy | @rhettinger, @ncoghlan, @pitrou, @vstinner, @ezio-melotti, @methane, @4kir4, @Julian, @berkerpeksag, @vadmium, @serhiy-storchaka, @jleedev, @matrixise, @gsnedders, @miss-islington |
| PRs | |
| Dependencies | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/ncoghlan'
closed_at = <Date 2016-09-10.10:24:12.011>
created_at = <Date 2013-05-05.13:10:30.725>
labels = ['type-feature', 'library', 'expert-unicode']
title = 'Autodetecting JSON encoding'
updated_at = <Date 2018-06-07.10:21:22.636>
user = 'https://github.com/serhiy-storchaka'bugs.python.org fields:
activity = <Date 2018-06-07.10:21:22.636>
actor = 'miss-islington'
assignee = 'ncoghlan'
closed = True
closed_date = <Date 2016-09-10.10:24:12.011>
closer = 'ncoghlan'
components = ['Library (Lib)', 'Unicode']
creation = <Date 2013-05-05.13:10:30.725>
creator = 'serhiy.storchaka'
dependencies = ['12892']
files = ['35258', '43513']
hgrepos = []
issue_num = 17909
keywords = ['patch']
message_count = 13.0
messages = ['188442', '218608', '218616', '218640', '218641', '230053', '273908', '275611', '275612', '275614', '318918', '318920', '318922']
nosy_count = 17.0
nosy_names = ['rhettinger', 'ncoghlan', 'pitrou', 'vstinner', 'ezio.melotti', 'cvrebert', 'methane', 'akira', 'Julian', 'python-dev', 'berker.peksag', 'martin.panter', 'serhiy.storchaka', 'jleedev', 'matrixise', 'gsnedders', 'miss-islington']
pr_nums = ['7366', '7474', '7475']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue17909'
versions = ['Python 3.6']
Activity
serhiy-storchaka commentedon May 5, 2013
RFC 4627 specifies a method to determine an encoding (one of UTF-8, UTF-16(BE|LE) or UTF-32(BE|LE)) of encoded JSON text. The proposed preliminary patch (it doesn't include the documentation yet) allows load() and loads() functions accept bytes data when it is encoded with standard Unicode encoding. Also accepted data with BOM (this doesn't specified in RFC 4627, but is widely used).
There is only one case where the method can give a misfire. Serialized string "\x00..." encoded in UTF-16LE may be erroneously detected as encoded in UTF-32LE. This case violates the two rules of RFC 4627: the string was serialized instead of a an object or an array, and the control character U+0000 was not escaped. The standard encoded JSON always detected correctly.
This patch requires "surrogatepass" error handler for utf-16/32 (see bpo-12892 and bpo-13916).
serhiy-storchaka commentedon May 15, 2014
All dependencies for this issue are resolved now.
Here is updated patch, synchronized with tip.
cvrebert commentedon May 15, 2014
You'll need to also update the "Character Encodings" subsection of the json docs.
4kir4 commentedon May 16, 2014
Both json standard (ECMA-404) [1] and the new json rfc 7159 [2] do not mention
the encoding detection.
[1] http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf
[2] https://tools.ietf.org/html/rfc7159#section-8.1
From the rfc:
Implementations MUST NOT add a byte order mark to the beginning of a
JSON text. In the interests of interoperability, implementations
that parse JSON texts MAY ignore the presence of a byte order mark
rather than treating it as an error.
cvrebert commentedon May 16, 2014
I agree that the state of encoding detection in the new RFC seems unclear, given that the old RFC prefaced the part about the encoding detection with:
But in the new RFC:
Thus,
There seems to have been a thread about encoding detection in the RFC 7159 working group, but I don't have the time to read through it all:
It eventually leads to a dedicated sub-thread:
vadmium commentedon Oct 27, 2014
If you adjusted the detect_encoding() logic according to Pete Cordell’s table at the bottom of <http://www.ietf.org/mail-archive/web/json/current/msg01959.html\>, it might work for standalone strings.
However since the RFC encourages UTF-8 for best interoperability, I wonder if any of this autodetection is necessary. It might be simpler to just assume UTF-8, or use the “utf-8-sig” codec. Or are there real cases where detecting UTF-16 or -32 would be useful?
matrixise commentedon Aug 30, 2016
Hi Serhiy,
I have reviewed your patch, it seems to be ok.
ncoghlan commentedon Sep 10, 2016
Having hit the json.loads() problem recently when porting a project to Python 3, I'm keen to see this land for 3.6.
Accodingly, assigning to myself to review and merge Serhiy's patch - if it proves necessary, we can tweak the details of the encoding detection during beta.
python-dev commentedon Sep 10, 2016
New changeset e9e1bf9ec2ac by Nick Coghlan in branch 'default':
Issue bpo-17909: Accept binary input in json.loads
https://hg.python.org/cpython/rev/e9e1bf9ec2ac
ncoghlan commentedon Sep 10, 2016
Thanks for tackling this Serhiy!
I removed bpo-13916 from the dependency list, as while that's a reasonable suggestion, I don't think this fix is conditional on that change.
methane commentedon Jun 7, 2018
New changeset bb6366b by INADA Naoki (Anthony Sottile) in branch 'master':
bpo-17909: Document that json.load can accept a binary IO (GH-7366)
bb6366b
miss-islington commentedon Jun 7, 2018
New changeset f38ace6 by Miss Islington (bot) in branch '3.7':
bpo-17909: Document that json.load can accept a binary IO (GH-7366)
f38ace6
miss-islington commentedon Jun 7, 2018
New changeset 21f2553 by Miss Islington (bot) in branch '3.6':
bpo-17909: Document that json.load can accept a binary IO (GH-7366)
21f2553