Make stdout and stderr truly unbuffered when using -u option #74589

Activity

1. serhiy-storchaka commented on May 19, 2017

   

   serhiy-storchaka

   on May 19, 2017

   MemberAuthor

   More actions

   In Python 2 when run the interpreter with the -u option the stdout and stderr streams are unbuffered. In Python 3 they become just line-buffered. This is because initially there was no way to create unbuffered text streams. But since Python 3.3 TextIOWrapper supports unbuffered output binary stream and accepts the write_through argument which switch off its own buffering.

   Proposed patch makes the stdout and stderr streams truly unbuffered when run with the -u option.
2. serhiy-storchaka

   added

   3.7 (EOL)end of lifeend of life

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   topic-IO

   type-featureA feature request or enhancementA feature request or enhancement

   on May 19, 2017
3. terryjreedy

   changed the title [-]Make stdout and stderr truly unbuffered when run with the -u option[/-] [+]Make stdout and stderr truly unbuffered when using -u option[/+] on May 19, 2017
4. vstinner commented on Jun 19, 2017

   

   vstinner

   on Jun 19, 2017

   Member

   More actions

   Oh, I like #1667

   "If write_through is True, calls to write() are guaranteed not to be buffered: any data written on the TextIOWrapper object is immediately handled to its underlying binary buffer."

   I didn't know write_through. It seems like it was introduced in Python 3.7: bpo-30526, commit 3c2817b.
5. vstinner commented on Jun 19, 2017

   

   vstinner

   on Jun 19, 2017

   Member

   More actions

   While I hope that users of the -u options expect the slow-down, would it be possible to benchmark it?

   For example, try to write setup.py content character by character to stdout using -u or not, into a TTY, into a pipe and/or a file. I'm curious if the line buffering vs really unbuffered has a significant impact on performances.
6. serhiy-storchaka commented on Jun 19, 2017

   

   serhiy-storchaka

   on Jun 19, 2017

   MemberAuthor

   More actions

   Writing separate lines:

   $ ./python -m timeit -s 'import sys' -s 'with open("setup.py") as f: s = f.readlines()' 'sys.stderr.writelines(s)' 2>/dev/null
   200 loops, best of 5: 1.07 msec per loop
   
   $ ./python -u -m timeit -s 'import sys' -s 'with open("setup.py") as f: s = f.readlines()' 'sys.stderr.writelines(s)' 2>/dev/null
   Unpatched:  50 loops, best of 5: 5.89 msec per loop
   Patched:    100 loops, best of 5: 3.32 msec per loop

   Writing separate characters:

   $ ./python -m timeit -s 'import sys' -s 'with open("setup.py") as f: s = list(f.read())' 'sys.stderr.writelines(s)' 2>/dev/null
   10 loops, best of 5: 30 msec per loop
   
   $ ./python -u -m timeit -s 'import sys' -s 'with open("setup.py") as f: s = list(f.read())' 'sys.stderr.writelines(s)' 2>/dev/null
   Unpatched:  5 loops, best of 5: 49.2 msec per loop
   Patched:    2 loops, best of 5: 137 msec per loop
7. vstinner commented on Jun 19, 2017

   

   vstinner

   on Jun 19, 2017

   Member

   More actions

   Hum. It has an huge impact on performances. Would it make sense to have two command line options to choose between unbuffered and line buffered?

   The C setvbuf() function uses these constants:

             _IONBF unbuffered
             _IOLBF line buffered
             _IOFBF fully buffered
   
8. serhiy-storchaka commented on Jun 19, 2017

   

   serhiy-storchaka

   on Jun 19, 2017

   MemberAuthor

   More actions

   Note that if output by lines, the patch speeds up the output! Actually the output is fast enough with buffering and without. The only slowdown is exposed when output by characters, but this is uncommon case. And if you output by characters (in case of drawing a progressbar or like), then perhaps you want the characters been displayed immediately, without buffering.

   See also bpo-13601.
9. serhiy-storchaka commented on Oct 4, 2017

   

   serhiy-storchaka

   on Oct 4, 2017

   MemberAuthor

   More actions

   If there is a need in making redirected stdout line-buffered, a new option can be added in separate issue.
10. serhiy-storchaka commented on Oct 4, 2017

    

    serhiy-storchaka

    on Oct 4, 2017

    MemberAuthor

    More actions

    New changeset 77732be by Serhiy Storchaka in branch 'master':
    bpo-30404: The -u option now makes the stdout and stderr streams totally unbuffered. (bpo-1667)
    77732be
11. serhiy-storchaka

    closed this as completedon Oct 4, 2017
12. berkerpeksag commented on Oct 13, 2017

    

    berkerpeksag

    on Oct 13, 2017

    Member

    More actions

    New changeset 7f58097 by Berker Peksag in branch 'master':
    bpo-28647: Update -u documentation after bpo-30404 (GH-3961)
    7f58097
13. ezio-melotti

    transferred this issue fromon Apr 10, 2022