Empty iterator with fake __len__ is rendered as a single bracket ] when using json's iterencode

@rhettinger

BPO	27613
Nosy	@rhettinger, @terryjreedy, @etrepum, @ezio-melotti, @bitdancer, @serhiy-storchaka
PRs	bpo-27613: Fix json.dump() for collections with overridden __bool__(). #1257

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-04-23.16:42:00.000>
created_at = <Date 2016-07-25.11:42:11.680>
labels = ['3.7', 'invalid', 'type-bug', 'library']
title = "Empty iterator with fake __len__ is rendered as a single bracket ] when using json's iterencode"
updated_at = <Date 2017-04-23.20:45:03.881>
user = 'https://bugs.python.org/altvod'

bugs.python.org fields:

activity = <Date 2017-04-23.20:45:03.881>
actor = 'terry.reedy'
assignee = 'none'
closed = True
closed_date = <Date 2017-04-23.16:42:00.000>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2016-07-25.11:42:11.680>
creator = 'altvod'
dependencies = []
files = []
hgrepos = []
issue_num = 27613
keywords = []
message_count = 14.0
messages = ['271249', '271250', '271251', '271252', '271254', '271255', '271258', '271259', '271272', '271641', '292124', '292172', '292173', '292185']
nosy_count = 8.0
nosy_names = ['rhettinger', 'terry.reedy', 'bob.ippolito', 'ezio.melotti', 'r.david.murray', 'SilentGhost', 'serhiy.storchaka', 'altvod']
pr_nums = ['1257']
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue27613'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7']

JSONEncoder.iterencode doesn't work with empty iterators correctly.
Steps:

Define an iterator that is recognized by json as a list (inherit from list and define nonzero __len__).
Use json.dump with data containing an empty iterator defined as described in step#1 (but doesn't generate any items)

Expected result: it should be rendered as an empty list: '[]'

Actual result: it is rendered as ']' (only the closing bracket)
interestingly enough this behavior is not reproduced when using the dumps function.
I tried other alternatives to the standard json module: simplejson, ujson, hjson
All of them work as expected in this case (both brackets are rendered).

Here is an example of the code that demonstrates this error (compares the results of the dump and dumps functions):

import json as json
import io

class EmptyIterator(list):
    def __iter__(self):
        while False:
            yield 1
    def __len__(self):
        return 1

def dump_to_str(data):
    return json.dumps(data)

def dump_to_file(data):
    stream = io.StringIO()
    json.dump(data, stream)
    return stream.getvalue()


data = {'it': EmptyIterator()}
print('to str: {0}'.format(dump_to_str(data)))
print('to file: {0}'.format(dump_to_file(data)))

This prints:
to str: {"it": []}
to file: {"it": ]}

Why does your __len__ method returns 1? Shouldn't it be 0 since this is an empty iterator? Changing it to zero seems to fix the "issue" too.

I can't do that if I don't know how many entries there will be ahead of time. In my real-life situation I'm fetching the data from a database not knowing how many entries I'll get before I actually get them (in the iterator). In most cases there are huge amounts of entries that take up too much memory - that's why I need to stream it. But sometimes the result set is empty - and that's when everything fails.

Actually, it does work with len = 0
even if the iterator is not empty. So, I guess that is a solution.
But still, I think the more correct way would be to make it work with > 0

My bad - it doesn't work with non-empty iterators if you set len to 0, so not a solution

The question is why are you defining __len__ if you don't know the size of your final object? Or at least, why are you starting with a potentially wrong initial value? This issue doesn't exist if you either don't define the method or return correct value.

If __len__ is not defined, then the iterator is considered empty and is always rendered as [] even if it really isn't empty

With streaming you never know the real length before you're done iterating.

Anyway, the fix really shouldn't be that complicated:
In _iterencode_list just yield the '[' instead of saving it to buf

If you break the invariants (in this case: a list has an accurate len) code that expects lists is only going to work by accident.

What you really want to do is define your own json encoder for your type. If that isn't possible for a streamed sequence of undefined length, then enhancing json's extension machinery to allow it would be a good feature request.

That said, could json's ability to handle this be improved? Possibly. I think we would accept a patch if it doesn't make the code more complicated. Ideally handling certain cases as "don't care" makes the code simpler, but that may or may not be the case here. Or it may come out as a consequence of the enhancement.

I am surprised that dumping to a string and to a file give different answers. This strikes me as a bug in any case. I would expect the the main difference would be file.write(chunk) versus temlist.append(chunk).

I am surprised that dumping to a string and to a file give different answers.

This is a difference between Python and C implementations.

An iterable with fake __len__ looks breaking the invariants, but if a collection with overridden __bool__() is considered as more legitimate, the proposed patch fixes Python implementation of json for that case.

Per PEP-8, the Python preferred-style is, "For sequences, (strings, lists, tuples), use the fact that empty sequences are false."

Yes: if not seq:
    if seq:

No: if len(seq):
    if not len(seq):

The Python libraries are not obliged to defend themselves against non-sensical types (i.e. defining an empty iterator as a subclass of list and returning a non-zero len).

I recommend leaving the code as-is and closing a "not a bug". In a way, this report is no more interesting than observing that a __hash__ that returns a random value on each call doesn't work well in a dictionary.

This looks as a strong argument to me. Thanks Raymond.

Thank you Raymond. What I missed before is a) the OP's misnamed EmptyIterator is an iterable (possibly non-empty) but not an iterator, empty or otherwise, and b) a sequence __len__ that lies is just a bug. (So is a iterator that does not yield the contents of a collection.) A non-0 length(lst) is a promise that lst[0] exists. Depending on this is routine. If json.encoder line 296, 'for value in lst:' were replaced by the following, which should be equivalent,
for i in range(len(list)):
value = lst[i]
the encoding would die with IndexError.

The intention that buggy code should not cause a crash was met in this case.

Navigation Menu

Repository navigation

Empty iterator with fake len is rendered as a single bracket ] when using json's iterencode #71800

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Empty iterator with fake __len__ is rendered as a single bracket ] when using json's iterencode #71800

Description

Activity

altvod commented on Jul 25, 2016

SilentGhost commented on Jul 25, 2016

altvod commented on Jul 25, 2016

altvod commented on Jul 25, 2016

altvod commented on Jul 25, 2016

SilentGhost commented on Jul 25, 2016

altvod commented on Jul 25, 2016

altvod commented on Jul 25, 2016

bitdancer commented on Jul 25, 2016

terryjreedy commented on Jul 29, 2016

serhiy-storchaka commented on Apr 22, 2017

rhettinger commented on Apr 23, 2017

serhiy-storchaka commented on Apr 23, 2017

terryjreedy commented on Apr 23, 2017

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Empty iterator with fake len is rendered as a single bracket ] when using json's iterencode #71800