Json.dump() bug when using generator #78031

Activity

1. biloup commented on Jun 13, 2018

   

   biloupmannequin

   on Jun 13, 2018

   MannequinAuthor

   More actions

   I use a class to write easily json when having generator.

   class StreamArray(list):
       def __init__(self, generator):
           super().__init__()
           self.generator = generator
       def __iter__(self):
           return self.generator
       def __len__(self):
           return 1

   Below a test comparing json.dump and json.dumps.

   >>> import json
   >>> class StreamArray(list):
   ...     def __init__(self, generator):
   ...         super().__init__()
   ...         self.generator = generator
   ...     def __iter__(self):
   ...         return self.generator
   ...     def __len__(self):
   ...         return 1
   ... 
   >>> g = (i for i in range(0))
   >>> json.dumps({"a": StreamArray(g)})
   '{"a": []}'
   >>> f = open("/tmp/test.json", "w+")
   >>> g = (i for i in range(0))
   >>> json.dump({"a": StreamArray(g)}, f)
   >>> f.close()
   >>> print(open("/tmp/test.json").read())
   {"a": ]}
   

   I don't know if it's me or if there is actually a problem, could you help me ?
2. biloup

   added

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Jun 13, 2018
3. ericvsmith commented on Jun 13, 2018

   

   ericvsmith

   on Jun 13, 2018

   Member

   More actions

   You should ask your question on this mailing list: https://mail.python.org/mailman/listinfo/python-list

   The bug tracker is not a place for asking how to use Python. If you actually find a bug in Python, you can re-open this issue. I do not believe that what you show here is a Python bug.

   Good luck!
4. ericvsmith

   closed this as completedon Jun 13, 2018
5. ericvsmith

   added

   invalid

   on Jun 13, 2018
6. doerwalter commented on Jun 13, 2018

   

   doerwalter

   on Jun 13, 2018

   Contributor

   More actions

   The problem here is that StreamArray lies about the length of the iterator. This confuses json.encoder._make_iterencode._iterencode_list(), (which is called by json.dump()), because it first does a check for "if not lst" and then assumes in the loop that it will be entered at least once.

   (Note that json.dumps() doesn't have that problem, because it calls JSONEncoder.encode() with _one_shot=True which leads to a totally different code path).

   We could declare that bug as "don't do that then", but the problem is easily solvable, because we can check whether the loop was entered. The attached patch should do the trick.

   An even better approach would IMHO be, that the encoder supports a special flag that enables JSON serialization of generators directly, so it's no longer required to masquerade generators as list
7. doerwalter

   reopened this on Jun 13, 2018
8. doerwalter

   removed

   invalid

   on Jun 13, 2018
9. serhiy-storchaka commented on Jun 13, 2018

   

   serhiy-storchaka

   on Jun 13, 2018

   Member

   More actions

   This is a duplicate of bpo-27613.
10. serhiy-storchaka

    closed this as completedon Jun 13, 2018
11. ezio-melotti

    transferred this issue fromon Apr 10, 2022