multiprocessing.Process depends on sys.stdout being open #72513

Activity

1. progval commented on Oct 1, 2016

   

   progvalmannequin

   on Oct 1, 2016

   MannequinAuthor

   More actions

   Hello,

   The following code:

   import sys
   import multiprocessing
   
   sys.stdout.close()
   
   def foo():
       pass
   
   p = multiprocessing.Process(target=foo)
   p.start()

   Crashes with:

   Traceback (most recent call last):
     File "foo.py", line 10, in <module>
       p.start()
     File "/usr/lib/python3.5/multiprocessing/process.py", line 105, in start
       self._popen = self._Popen(self)
     File "/usr/lib/python3.5/multiprocessing/context.py", line 212, in _Popen
       return _default_context.get_context().Process._Popen(process_obj)
     File "/usr/lib/python3.5/multiprocessing/context.py", line 267, in _Popen
       return Popen(process_obj)
     File "/usr/lib/python3.5/multiprocessing/popen_fork.py", line 17, in __init__
       sys.stdout.flush()
   ValueError: I/O operation on closed file.

   This bug has been reported to me on a daemonized program (written prior to PEP-3143).
2. progval

   added

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   on Oct 1, 2016
3. tiagoantao commented on Oct 2, 2016

   

   tiagoantaomannequin

   on Oct 2, 2016

   Mannequin

   More actions

   I made a small patch for this. This is the first time I submit one, so please be careful with this...
4. terryjreedy commented on Oct 7, 2016

   

   terryjreedy

   on Oct 7, 2016

   Member

   More actions

   Patch looks plausible to me. The flush calls definitely need to be guarded.
5. terryjreedy

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Oct 7, 2016
6. tiagoantao commented on Oct 7, 2016

   

   tiagoantaomannequin

   on Oct 7, 2016

   Mannequin

   More actions

   This is the first patch that I am submitting. More than willing to do any changes deemed necessary...
7. tiagoantao

   added

   3.7 (EOL)end of lifeend of life

   on May 3, 2017
8. pitrou commented on Oct 21, 2017

   

   pitrou

   on Oct 21, 2017

   Member

   More actions

   In bpo-31804, we see that sys.stdout and sys.stderr can sometimes simply be None, so we must check for that too.
9. pitrou commented on Oct 22, 2017

   

   pitrou

   on Oct 22, 2017

   Member

   More actions

   New changeset daeefd2 by Antoine Pitrou in branch 'master':
   bpo-28326: Fix multiprocessing.Process when stdout and/or stderr is closed or None. (bpo-4073)
   daeefd2
10. pitrou commented on Oct 22, 2017

    

    pitrou

    on Oct 22, 2017

    Member

    More actions

    New changeset 34ef6da by Antoine Pitrou in branch '3.6':
    [3.6] bpo-28326: Fix multiprocessing.Process when stdout and/or stderr is closed or None. (GH-4073). (bpo-4075)
    34ef6da
11. pitrou commented on Oct 22, 2017

    

    pitrou

    on Oct 22, 2017

    Member

    More actions

    This is now fixed in 3.6 and 3.7. Closing!
12. pitrou

    closed this as completedon Oct 22, 2017
13. ezio-melotti

    transferred this issue fromon Apr 10, 2022
14. betterlch commented on Sep 14, 2026

    

    betterlch

    on Sep 14, 2026

    Last edited by betterlch

    More actions

    Hello , I met some error when using multiprocessing

    Process.start() raises BrokenPipeError: [Errno 32] Broken pipe when stdout is a pipe with a dead read end — _flush_std_streams() does not catch OSError

    def _flush_std_streams():
        try:
            sys.stdout.flush()
        except (AttributeError, ValueError):   # BrokenPipeError (an OSError) escapes
            pass
        ... 

    If sys.stdout holds unflushed buffered data and its pipe's read end is gone, sys.stdout.flush() raises BrokenPipeError, which propagates out of Process.start().

    import os, sys, multiprocessing
    
    def target():
        pass
    
    if __name__ == "__main__":
        # Simulate a scheduler-launched process whose stdout reader exited.
        r, w = os.pipe()
        os.dup2(w, 1)                    # fd 1 -> the pipe
        os.close(r)                      # read end gone -> broken pipe
        os.close(w)                      # fd 1 keeps the write end alive
        sys.stdout = os.fdopen(1, "w")   # block-buffered (pipe, not a tty)
    
        sys.stdout.write("unflushed data")   # stays in the userspace buffer
    
        p = multiprocessing.Process(target=target)
        p.start()   # raises BrokenPipeError
        p.join()

    How we use: the process was launched over SSH (paramiko exec_command), and the caller closed the channel without draining output