popen spawned process may not write to stdout under windows #45707

Activity

1. pmezard commented on Oct 31, 2007

   

   pmezardmannequin

   on Oct 31, 2007

   MannequinAuthor

   More actions

   Let child.py be:
   """
   import sys

   sys.stdout.write('1:stdout\n')
   sys.stdout.flush()
   sys.stderr.write('2:stderr\n')
   sys.stderr.flush()
   sys.stdout.write('3:stdout\n')
   sys.stdout.flush()
   """

   and parent.py:
   """
   import os

   cmd = 'python child.py'
   for l in os.popen(cmd):
       print l,
   """

   Then running it:
   """

   python parent.py
   1:stdout

   python
   Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit
   (Intel)] on win32
   """

   I would have expected at least:
   """
   1:stdout
   3:stdout
   """
   to be output, which actually happens if the stderr is nullified like
   "python child.py 2>nul" in parent.py call.

   Am I wrong ?
2. pmezard

   added

   OS-windows

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Oct 31, 2007
3. ggenellina commented on Nov 7, 2007

   

   ggenellinamannequin

   on Nov 7, 2007

   Mannequin

   More actions

   (I think the title you meant was
   "popen spawned process may not
   write to stderr under windows")

   The child is dying with IOError:
   [Errno 22] Invalid argument
   at the sys.stderr.flush() call.

   Neither the docs for os.popen nor
   the Linux man page for popen(3)
   say that stderr is redirected, so
   one would expect the handle to be
   inherited; the IOError looks like
   a bug.
   Try using os.popen4 or
   popen2.popen4 or -the recommended
   choice- the subprocess module.
   Using the latter, this is the
   modified parent.py:

   """
   import subprocess
   cmd = 'python child.py'
   p = subprocess.Popen(cmd,
   stdout=subprocess.PIPE)
   for line in p.stdout:
   print ">>>", line,
   print p.wait()
   """

   and this is the output, as
   expected:

   """
   2:stderr
   >>> 1:stdout
   >>> 3:stdout
   0
   """

   Note the 2:stderr line lacking the

   >>, because it was printed
   directly by the child process onto
   the stderr handle inherited from
   its parent.
4. pythonmeister commented on Nov 8, 2007

   

   pythonmeistermannequin

   on Nov 8, 2007

   Mannequin

   More actions

   the popen call does not redirect stderr.
   If you do something like 2>null (windows) or 2>/dev/null (*nix) it will
   _never_ get printed.
   If you want to have stderr & stdout getting in via popen and thus stdout,
   under *nix and windows you would do that:

   command 2>&1

   It is not popen to blame.
   See this for reference:
   http://netbsd.gw.com/cgi-bin/man-cgi?popen++NetBSD-current
5. pmezard commented on Nov 8, 2007

   

   pmezardmannequin

   on Nov 8, 2007

   MannequinAuthor

   More actions

   pythonmeister: I never expected stderr to be redirected, just *all
   stdout* to be captured. But...

   gagenellina: you are completely right about the failure. Still, this
   issue happened with a real world application written in C, and
   redirecting manually stderr to :NUL: solved it unexpectedly. I did not
   expect spawned process behaviour to differ when its stderr is being
   redirected.
6. jafo commented on Mar 17, 2008

   

   jafomannequin

   on Mar 17, 2008

   Mannequin

   More actions

   We've discussed this at the PyCon sprints, and here's the concensus:

   os.popen inherits the parents stderr, and on Windows there is not an
   existing valid stderr by default. So the parent should, to be
   compatible with the Windows environment, create stderr or use popen2.

   However, popen is deprecated. The best solution would be to use
   subprocess module which is defined to do the right thing in this case.
   popen is not.

   Because popen is deprecated, we are going to leave this behavior and
   documentation as it is.
7. birkenfeld commented on Jul 16, 2008

   

   birkenfeld

   on Jul 16, 2008

   Member

   More actions

   Did you want to close this, Sean?
8. birkenfeld

   closed this as completedon Dec 5, 2008
9. ezio-melotti

   transferred this issue fromon Apr 10, 2022