extend json.tool --json-lines to ignore empty rows #90087

Activity

1. ZeeD commented on Nov 29, 2021

   

   ZeeDmannequin

   on Nov 29, 2021

   Last edited by StanFromIreland

   MannequinAuthor

   More actions

   It would be useful to let json.tool support empty rows during handling of json lines
   generally speaking, this tolerance is already present in parsers like srsly and jsonlines

   actual behavior:

   \# happy scenario
   $ echo -e '{"foo":1}\\n{"bar":2}' | python3.10 -mjson.tool --json-lines
   {
       "foo": 1
   }
   {
       "bar": 2
   }
   

   \# spurious EOL at EOF
   $ echo -e '{"foo":1}\\n{"bar":2}\\n' | python3.10 -mjson.tool --json-lines
   {
       "foo": 1
   }
   {
       "bar": 2
   }
   Expecting value: line 2 column 1 (char 1)
   

   \# two groups of "rows" in jsonl \<- my current usecase
   $ echo -e '{"foo":1}\\n\\n{"bar":2}' | python3.10 -mjson.tool --json-lines
   {
       "foo": 1
   }
   Expecting value: line 2 column 1 (char 1)
   

   my desired outcome is to preserve the EOLs, so to have something like:

   \# happy scenario
   $ echo -e '{"foo":1}\\n{"bar":2}' | python3.10 -mjson.tool --json-lines
   {
       "foo": 1
   }
   {
       "bar": 2
   }
   

   \# spurious EOL at EOF
   $ echo -e '{"foo":1}\\n{"bar":2}\\n' | python3.10 -mjson.tool --json-lines
   {
       "foo": 1
   }
   {
       "bar": 2
   }
   
   

   \# two groups of "rows" in jsonl
   $ echo -e '{"foo":1}\\n\\n{"bar":2}' | python3.10 -mjson.tool --json-lines
   {
       "foo": 1
   }
   
   {
       "bar": 2
   }
   
2. ZeeD

   added

   3.10 (EOL)end of lifeend of life

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   type-featureA feature request or enhancementA feature request or enhancement

   on Nov 29, 2021
3. AlexWaygood commented on Dec 7, 2021

   

   AlexWaygood

   on Dec 7, 2021

   Member

   More actions

   I am changing the "version" field to 3.11, as enhancement proposals are generally only considered for unreleased versions of Python.
4. AlexWaygood

   added

   3.11only security fixesonly security fixes

   and removed

   3.10 (EOL)end of lifeend of life

   on Dec 7, 2021
5. serhiy-storchaka commented on Dec 7, 2021

   

   serhiy-storchaka

   on Dec 7, 2021

   Member

   More actions

   Both JSON Lines (https://jsonlines.org/) and Newline Delimited JSON (http://ndjson.org/) formats require that Each Line is a Valid JSON Value.

   If you want to ignore empty lines you can filter them out with sed /^$/d.
6. ZeeD commented on Dec 8, 2021

   

   ZeeDmannequin

   on Dec 8, 2021

   MannequinAuthor

   More actions

   My final goal is to preserve the empty lines - I think I can do some bash magic, but maybe something more complex that a simple sed call.

   FWIW on https://jsonlines.org/#line-separator-is-n I see "The last character in the file may be a line separator, and it will be treated the same as if there was no line separator present.".
   And on https://github.com/ndjson/ndjson-spec#32-parsing I see "The parser MAY silently ignore empty lines, e.g. \n\n. This behavior MUST be documented and SHOULD be configurable by the user of the parser.".

   While I get this choice can be "on a grey area", I think this is a known "dialect" of the jsonl specs.
7. ambv commented on Dec 10, 2021

   

   ambv

   on Dec 10, 2021

   Contributor

   More actions

   I agree with Serhiy that in general the fact that json.tool crashes here is useful: similarly to an exception in Python code, it can inform the user that some data they feed to json.tool is invalid.

   At the same time, I find it a bit obnoxious that the current implementation doesn't allow for the *final character* of the input to be a newline (or "characters" in case of \r\n... but it should still only ignore *a single effective newline*).

   I mean, if the user starts spewing newlines in the middle of the file... that might easily be an error. If the file ends with 5 empty lines, that might easily be an error. But, if the file really is:

   {'line': 1, 'data': ...}\n
   {'line': 2, 'data': ...}\n
   {'line': 3, 'data': ...}\n

   I think that should be pragmatically accepted by json.tool, especially since many text editors now add newline characters at file ends.
8. serhiy-storchaka commented on Dec 11, 2021

   

   serhiy-storchaka

   on Dec 11, 2021

   Member

   More actions

   The current implementation allows for the final character of the input to be a newline. It does not allow double newlines. In the original example

   echo -e '{"foo":1}\n{"bar":2}\n'

   the echo command adds a newline to the output (which already contains the trailing newline), so the result ends with two newlines. Use option -n to disable adding newline in echo.

   I afraid that if we add support of empty lines, soon we will get requests for supporting comments, encoding cookies, single-quote strings, non-quoted keys, hexadecimal integers and other possible JSON extensions.
9. ezio-melotti

   transferred this issue fromon Apr 10, 2022
10. serhiy-storchaka

    added this to @serhiy-storchaka's project python-jsonand @serhiy-storchaka's project python-json and removed this from @serhiy-storchaka's project python-jsonon May 8, 2022
11. serhiy-storchaka

    added this to JSON issueson May 19, 2022
12. StanFromIreland

    removed

    3.11only security fixesonly security fixes

    on Aug 10, 2025