Repository navigation
Python interactive console doesn't use sys.stdin for input #61820
Description
| BPO | 17620 |
|---|---|
| Nosy | @gvanrossum, @birkenfeld, @pfmoore, @ncoghlan, @pitrou, @benjaminp, @merwok, @zooba |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2013-04-02.16:59:14.835>
labels = ['type-feature', '3.8', '3.9', '3.10']
title = "Python interactive console doesn't use sys.stdin for input"
updated_at = <Date 2021-03-15.21:01:39.375>
user = 'https://bugs.python.org/Drekin'bugs.python.org fields:
activity = <Date 2021-03-15.21:01:39.375>
actor = 'eryksun'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2013-04-02.16:59:14.835>
creator = 'Drekin'
dependencies = []
files = []
hgrepos = []
issue_num = 17620
keywords = []
message_count = 29.0
messages = ['185848', '186121', '186553', '186576', '186580', '193815', '221176', '221179', '223414', '224312', '224313', '224330', '224334', '224338', '224396', '224397', '226021', '226098', '226099', '226100', '226126', '226140', '226933', '234439', '242173', '255549', '272681', '272683', '275242']
nosy_count = 10.0
nosy_names = ['gvanrossum', 'georg.brandl', 'paul.moore', 'ncoghlan', 'pitrou', 'benjamin.peterson', 'eric.araujo', 'tshepang', 'Drekin', 'steve.dower']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue17620'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']
Activity
Drekin commentedon Apr 2, 2013
The Python interactive console actually doesn't use sys.stdin but standard C stdin for input. Is there any reason for this? Why it then uses its encoding attribute? (Assigning sys.stdin something, that doesn't have encoding attribute freezes the interpreter.) If anything, wouldn't it make more sense if it used sys.__stdin__.encoding instead of sys.stdin? sys.stdin is intended to be set by user (it affects input() and code.inpterrupt() which tries to minic standard interactive console).
Drekin commentedon Apr 6, 2013
Sorry for typos.
• interactive console doesn't use sys.stdin for input, why?
• it uses sys.stdin.encoding, shouldn't it rather use sys.__stdin__.encoding if anything?
• input() and hence code.interact() uses sys.stdin
pitrou commentedon Apr 11, 2013
Modules/main.c calls PyRun_AnyFileFlags(stdin, "<stdin>", ...). At this point, sys.stdin *is* the same as C stdin by construction, so I'm not sure how you came to encounter the issue.
However, it's also true that if you later redirect sys.stdin, it will be ignored and the original C stdin (as passed to PyRun_InteractiveLoopFlags) will continue to be used. On the other hand, the input() implementation has dedicated logic to find out whether sys.stdin is the same as C stdin.
(by the way, the issue should also apply to 2.7)
Assuming the previous bug gets fixed, then no :-)
Drekin commentedon Apr 11, 2013
I encountered it when I changed sys.stdin at runtime (I thought it was a supported feature) to affect the interactive console, see http://bugs.python.org/issue1602 .
pitrou commentedon Apr 11, 2013
Ok, I guess it would need a new API (PyRun_Stdio()?) to run the interactive loop from sys.stdin, rather than from a fixed FILE*.
Drekin commentedon Jul 28, 2013
Is there any chance the API will be added and used by python.exe?
ncoghlan commentedon Jun 21, 2014
Steve, another one to look at in the context of improving the Unicode handling situation at the Windows command prompt.
zooba commentedon Jun 21, 2014
Thanks Nick, but this has a pretty clear scope that may help the Unicode situation in cmd but doesn't directly relate to it.
Drekin commentedon Jul 18, 2014
There is still the serious inconsistency that the
sys.stdinis not used for input by interactive loop but its encoding is. So if I replacesys.stdinwith a custom object with its ownencodingattribute, the standard interactive loop tries to use this encoding which may result in an exception on any input.gvanrossum commentedon Jul 30, 2014
Is this at all related to the use of GNU readline?
10 remaining items
vstinner commentedon Aug 30, 2014
The Python parser works well with UTF8. If you know the encoding, decode
from your encoding and encode to UTF8. You should pass the UTF8 flag to the
parser.
Drekin commentedon Aug 30, 2014
Antoine Pitrou: I understand. It would be nice to have that new Python string based readline hook. Its default implementation could be to call PyOS_Readline and decode the bytes using sys.stdin.encoding (as the tokenizer currently does). Tokenizer then woudn't need to decode if it called the new hook.
Victor Stinner: I'm going to try the approach of reencoding my stream to UTF-8. So then my UTF-16-LE encoded stream is decoded, then encoded to UTF-8, interpreted as null-terminated *char, which is returned to the tokenizer, which again decodes it and encodes to UTF-8. I wonder if the last step could be short-circuited. What is this UTF8 flag to Python parser? I couldn't find any information.
Drekin commentedon Sep 15, 2014
I have found another problem. PyOS_Readline can be called from two different places – from Parser/tokenizer.c:tok_nextc (by REPL), which uses sys.stdin.encoding to encode prompt argument, and from Python/bltinmodule.c:builtin_input_impl (by input() function), which uses sys.stdout.encoding. So readline hook cannot be implemented correctly if sys.stdin and sys.stdout don't have the same encoding.
Either the tokenizer should have two encodings – one for input and one for output - or better no encoding at all and should use Python string based alternative to PyOS_Readline, which could be added.
Drekin commentedon Jan 21, 2015
Unfortunately, I have little or no experience with Python C code and I even don't have a C compiler installed so I cannot experiment. I'll just put my ideas how to solve this here.
• Add sys.__readlinehook__ attribute, which can be set to a function taking a prompt string and returing a line.
• Add C function PyOS_UnicodeReadline (possibly with a better name) which has the same signature as sys.__readlinehook__ (in contrast with the signature of PyOS_Readline). If sys.__readlinehook__ is set, call it; otherwise encode the prompt string using stdout encoding and delegate to PyOS_Readline and decode the string returned using stdin encoding.
• Change the tokenizer and the implementation of input() so it uses PyOS_UnicodeReadline rather than PyOS_Readline.
This would solve the problem that utf-16 encoded string cannot be given to the tokenizer and also would bypass the silent assumption that stdin and stdout encodings are the same. Also, readline hook could be easily set from Python code – no need for ctypes. The package pyreadline could use this. Also, the issue bpo-1602 could be then solved just by changing sys.std* streams and providing a trivial sys.__readlinehook__ delegating to sys.stdout.write and sys.stdin.readline.
Drekin commentedon Apr 28, 2015
Note that under status quo PyOS_Readline is called from two places: the tokenizer during an interactive session and the builtin function input. The tokenizer passes promptstring encoded in sys.stdin.encoding while input() passes promtstring encoded in sys.stdout.encoding, so it is not possible to implement a readline hook correctly in the case the encodings are different. This might be considered a bug.
Drekin commentedon Nov 28, 2015
I've formulated a proposal regarding this issue: https://mail.python.org/pipermail/python-dev/2015-November/142246.html . Does it make sense?
zooba commentedon Aug 14, 2016
I'm working on this as part of my fix for bpo-1602. Not yet sure how this will come out - compatibility with GNU readline seems to be the biggest issue, as if we want to keep that then we can't allow embedded '\0' in the encoded text (i.e. UTF-16 cannot be used, which implies that sys.stdin.encoding cannot always be used directly).
Adding __readlinehook__ as an alternative may be feasible, but a decent amount of work given how we call into the current readline implementation. Unfortunately, it looks like detecting when a readline hook has been added is going to involve significant changes to the tokenizer, which I really don't want to do.
The easiest approach wrt bpo-1602 seems to be to special case the console by reencoding from utf-16-le to utf-8 and forcing the encoding in the tokenizer to utf-8 (instead of sys.stdin.encoding) in this case. I'll start here so that at least we can parse Unicode from the interactive prompt.
Drekin commentedon Aug 14, 2016
We don't need to detect the presence of readline hook, it may be so that there is always a readline hook. Whenever we have interactive stdio, and so PyOS_Readline is called, the new proposed API PyIO_Readline would be called instead. This would return Unicode str Py_Object*, so the result can be directly returned by input() and should be somehow encoded afterwards by the tokenizer (these are the only consumers of PyOS_Readline).
We may even leave the tokenizer alone and redefine PyOS_Readline as a wrapper of PyIO_Readline, having full control of the encoding process there. So it would be enough to set up the tokenizer with UTF-8 encoding despite the fact that sys.std*.encoding would be UTF-16.
(I hope that if the tokenizer was desiged nowdays, it would operate on strings rather than bytes so there won't be any encoding problems at all.)
Also, third parties would benefit from sys.readlinehook – at least win_unicode_console and pyreadline would just set the attribute rather than messing with ctypes.
zooba commentedon Sep 9, 2016
Unassigning this. I meant to close it with another fix, but that would be wrong as we really ought to keep this open until we solve it properly. All I've done is make it use the right APIs on Windows, but we still don't handle it properly when we change stdin.