Repository navigation
idle3 shell os.system swallows shell command output #56029
Description
| BPO | 11820 |
|---|---|
| Nosy | @terryjreedy, @ned-deily, @serwy, @4kir4, @serhiy-storchaka, @pppery |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2011-04-10.07:45:39.292>
labels = ['interpreter-core', 'expert-IDLE', 'type-bug', '3.9']
title = 'idle3 shell os.system swallows shell command output'
updated_at = <Date 2019-09-19.18:39:22.828>
user = 'https://bugs.python.org/Thekent'bugs.python.org fields:
activity = <Date 2019-09-19.18:39:22.828>
actor = 'terry.reedy'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['IDLE', 'Interpreter Core']
creation = <Date 2011-04-10.07:45:39.292>
creator = 'Thekent'
dependencies = []
files = ['36968']
hgrepos = []
issue_num = 11820
keywords = ['patch']
message_count = 21.0
messages = ['133452', '133453', '133493', '133873', '133877', '133881', '133885', '133886', '161717', '161720', '161724', '161732', '161737', '229649', '229651', '229683', '229686', '229700', '231023', '231056', '297921']
nosy_count = 7.0
nosy_names = ['terry.reedy', 'ned.deily', 'roger.serwy', 'akira', 'Thekent', 'serhiy.storchaka', 'ppperry']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue11820'
versions = ['Python 3.9']
Activity
ned-deily commentedon May 27, 2012
I disagree. I think you are confusing Python's sys.std* objects with the std* file descriptors of a process. The important points here are that IDLE is effectively substituting its rpcproxy for the std* file descriptors of the IDLE shell process that it creates *and* that, in POSIX world, a forked subprocess is required to inherit copies of the file descriptors (in particular, sys.std*) of its parent (the IDLE shell subprocess). So, if the IDLE shell subprocess displays the output written to the stdout and stderr in the Python Shell window and directs typed-in input from that window to the stdin file descriptor, it is reasonable to assume that the same thing should happen by default to processes launched within that IDLE shell subprocess, be they by os.system or by subprocess.Popen. subprocess.Popen lets you substitute sys* file associations when creating the subprocess but that is irrelevant for the default case. The default for subprocess.Popen is exactly the same as os.system: "With the default settings of None, no redirection will occur; the child’s file handles will be inherited from the parent."
terryjreedy commentedon May 27, 2012
I thought I was un-confusing sys.std* Python objects (such as idlelib.rpc.RPCProxy) from std* integer file descriptors, whereas you seem to say they are the same: "copies of the file descriptors (in particular, sys.std*)". The rest of your message seems to imply that subprocess.check_output should not work, but it does, at least on Windows. And subprocess *is* a recommended replacement for os.system.
Roger, do you have any thoughts on this issue?
ned-deily commentedon May 27, 2012
I didn't mean to imply anything about subprocess.check_output(). It works as expected, that is by explicitly intercepting anything written to the stdout of the forked subprocess and returning it as the call result. What doesn't work is if you try a default subprocess.call(), for example. The results are the same as os.system and for the same reason. In a Posix terminal session:
That's confusing to users. IDLE should handle that.
serwy commentedon May 27, 2012
It might be possible to fix this problem by creating pipes to redirect stdout and stderr to the text widget in PyShell.
The ShellWindow.py demo in Demo/tkinter/guido may be helpful.
pppery commentedon Oct 18, 2014
The same lack of output occurs from processes started via the multiprocessing module.
terryjreedy commentedon Oct 18, 2014
bpo-22664 (closed as duplicate of this) has ppperry's multiprocessing test example.
serhiy-storchaka commentedon Oct 19, 2014
Thank you Roger for your suggestion. Here is a patch which handless subprocess output (it is more complicated than Guido's demo because we need Unicode strings, not bytes).
4kir4 commentedon Oct 19, 2014
It looks like the issue can be reduced to whether or not to show this
output:
assuming sys.__stdout__.fileno() == fileno(stdout) == 1.
If "should we see this in idle?" is shown in idle then the output that
writes to the inherited C stdout such as os.system('ls'),
subprocess.call('ls') will also be shown automatically.
And in reverse If "should we see this in idle?" should not be shown
then other C stdout output such as produced by os.system('ls') should
not be redirected too. The output written to the same file should be
shown/not shown consistently.
ned-deily commentedon Oct 19, 2014
A quick smoke test of the patch (solely on OS X) looks like it works pretty well. Not surprisingly, it doesn't fix anything when using the deprecated -n ("no subprocess") option but that's not a problem. One more serious issue is that, depending on the length of the output produced, the IDLE shell prompt ('>>>') for the next command may appear out of sync, interspersed within the displayed stdout rather than at the end. And, yes, the os.write(1, ...) to stdout in the same process is an equivalent way to look at the issue and the patch seems to handle that as well.
Nice work, Serhiy!
serhiy-storchaka commentedon Nov 11, 2014
What do you think about this patch Terry? [Edit: on bpo-11820]
terryjreedy commentedon Nov 12, 2014
Currently, when Idle is started from a command line or console interpreter, "import os; os.system('dir')" produces the listing in the console window, as expected.
Reading the patch, it *augments* one-channel socket communication, which properly* combines and serializes stdout and stderr messages, with two-channel pipe communication. This is rather messy and leads to inter-mixed displays, as reported. If sensibly possible, the process stdout/err (fileno 1&2) written to by the os functions in question should forward bytes to the idle process via the rpc channel, properly labelled as stdout or stderr.
Issue bpo-18823 is about *replacing* the socket with pipes (whether using subprocess or multi-processing or whatever.). (There should still be just one channel, not two, from user process to Idle process to keep the rpc protocol in control of what the Idle process receives.) About last August, Roger sent at least a few of us somewhat large proof of concept patch, which I have not reviewed yet.
The patch uses Tk.createfilehandler, which seems not to exist on Windows, so I cannot review further.
File "F:\Python\dev\34\lib\tkinter\init.py", line 1932, in __getattr__
return getattr(self.tk, attr)
AttributeError: 'tkapp' object has no attribute 'createfilehandler'
File "F:\Python\dev\34\lib\tkinter\init.py", line 1932, in __getattr__
return getattr(self.tk, attr)
AttributeError: 'tkapp' object has no attribute 'createfilehandler' File "F:\Python\dev\34\lib\tkinter\init.py", line 1932, in __getattr__
return getattr(self.tk, attr)
AttributeError: 'tkapp' object has no attribute 'createfilehandler'
terryjreedy commentedon Jul 8, 2017
Closed bpo-13220 as duplicate of this.
terryjreedy commentedon Sep 16, 2026
#139383, not getting subinterpreter stdout, closed as a duplicate of this