Repository navigation
subprocess seems to use local encoding and give no choice #50385
Description
| BPO | 6135 |
|---|---|
| Nosy | @gpshead, @amauryfa, @ncoghlan, @pitrou, @vstinner, @mark-summerfield, @merwok, @bitdancer, @mightyiam, @andyclegg, @cjerdonek, @vadmium, @eryksun, @zooba, @davispuh |
| PRs | subprocess.check_output() doc to mention changes in 3.6 #5564 |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/zooba'
closed_at = <Date 2016-09-07.03:18:30.672>
created_at = <Date 2009-05-28.07:08:42.275>
labels = ['type-feature', 'library']
title = 'subprocess seems to use local encoding and give no choice'
updated_at = <Date 2018-02-07.02:11:33.939>
user = 'https://github.com/mark-summerfield'bugs.python.org fields:
activity = <Date 2018-02-07.02:11:33.939>
actor = 'gregory.p.smith'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2016-09-07.03:18:30.672>
closer = 'steve.dower'
components = ['Library (Lib)']
creation = <Date 2009-05-28.07:08:42.275>
creator = 'mark'
dependencies = []
files = ['14293', '14294', '14295', '28760', '44386', '44389', '44398', '44404', '44408']
hgrepos = []
issue_num = 6135
keywords = ['patch']
message_count = 66.0
messages = ['88466', '89094', '89293', '89322', '89325', '89332', '89333', '97090', '97092', '97093', '111181', '111466', '123020', '123024', '148025', '148066', '148484', '148494', '148495', '148496', '148498', '168191', '168213', '180157', '265785', '265793', '265812', '265822', '274467', '274468', '274469', '274481', '274489', '274492', '274497', '274509', '274510', '274517', '274518', '274519', '274561', '274562', '274566', '274574', '274642', '274644', '274646', '274648', '274649', '274651', '274653', '274665', '274699', '274736', '274737', '282291', '282294', '282298', '304049', '304060', '304118', '304119', '304126', '311756', '311757', '311760']
nosy_count = 20.0
nosy_names = ['gregory.p.smith', 'amaury.forgeotdarc', 'ncoghlan', 'pitrou', 'vstinner', 'mark', 'eric.araujo', 'segfaulthunter', 'Arfrever', 'r.david.murray', 'srid', 'mightyiam', 'andrewclegg', 'chris.jerdonek', 'python-dev', 'martin.panter', 'eryksun', 'steve.dower', 'berwin22', 'davispuh']
pr_nums = ['5564', '5572', '5573']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue6135'
versions = ['Python 3.6']
Activity
mark-summerfield commentedon May 28, 2009
When I start a process with subprocess.Popen() and pipe the stdin and
stdout, it always seems to use the local 8-bit encoding.
I tried setting process.stdin.encoding = "utf8" and the same for stdout
(where process is the subprocess object), but to no avail.
I also tried using shell=True since on Mac, Terminal.app is fine with
Unicode, but that didn't work.
So basically, I have programs that output Unicode and run fine on the
Mac terminal, but that cannot be executed by subprocess because
subprocess uses the mac_roman encoding instead of Unicode.
I wish it were possible to specify the stdin and stdout encoding that is
used; then I could use the same one on all platforms. (But perhaps it is
possible, and I just haven't figured out how?)
srid commentedon Jun 8, 2009
Related discussion thread: https://answers.launchpad.net/bzr/+question/63601
amauryfa commentedon Jun 12, 2009
I propose to add two parameters (encoding, error) to the
subprocess.Popen function.
I'll try to come with a patch.
segfaulthunter commentedon Jun 13, 2009
I wrote a patch to add encoding and error to subprocess.Popen in Python
2.7 (trunk).
segfaulthunter commentedon Jun 13, 2009
Cosmetic update.
pitrou commentedon Jun 13, 2009
Two things:
errorsfor consistency with open()and TextIOWrapper(), not
errorsegfaulthunter commentedon Jun 13, 2009
Should we also cover the unusual case where stdout, stderr and stdin
have different encodings, because now we are assuming the are all the same.
mark-summerfield commentedon Dec 31, 2009
I agree with Florian Mayer that the encoding handling should be
stream-specific. You could easily be reading the stdout of some third
party program that uses, say, latin1, but want to do your own output in,
say, utf-8.
One solution that builds on what Amaury Forgeot d'Arc has done (i.e.,
the encoding and errors parameters) by allowing those parameters to
accept either a single string (as now), or a dict with keys 'stdin',
'stdout', 'stderr'. Of course it is possible that the client might not
specify all the dict's keys in which case those would use the normal
default (local 8-bit etc.)
amauryfa commentedon Dec 31, 2009
I don't understand. How is the subprocess stdout related to the main
program output?
Stream-specific encoding could be useful for subprocesses that expect
latin-1 from stdin but write utf-8 to stdout. I'm not sure we should
support this.
mark-summerfield commentedon Dec 31, 2009
On Thu, Dec 31, 2009 at 1:30 PM, Amaury Forgeot d'Arc
<report@bugs.python.org> wrote:
Yes, you're right.
(What I had in mind was a scenario where you read one process's stdout
and wrote to another process's stdin; but of course using your errors
& encoding arguments this will work because there'll be two separate
process objects each of which can have its encoding and errors set
separately.)
BreamoreBoy commentedon Jul 22, 2010
Ran new unit test before and after patching subprocess on Windows Vista against 3.1 debug maintenance release, all ok apart from this at end of latter.
File "test\test_subprocess.py", line 568, in test_encoded_stderr
self.assertEqual(p.stderr.read(), send)
AssertionError: 'ï[32943 refs]\r\n' != 'ï'
I'm sure I've seen a ref to this somewhere, can anyone remember where?
BreamoreBoy commentedon Jul 24, 2010
In 2.7 and py3k test_subprocess has a class BaseTestCase which has a assertStderrEqual method. These don't exist in 3.1. I believe that the py3k code will need to be backported to 3.1. Can this be done on this issue, or do we need a new one to keep things clean?
vstinner commentedon Dec 1, 2010
About the topic:
I don't see how you can get unicode from a process only using subprocess. But with Python 3, you can get unicode if you set universal_newlines option to True.
So for Python 2, it's a new feature (get unicode), and for Python 3, it's a new option to specify the encoding. The title should be changed to something like "subprocess: add an option to specify stdin, stdout and/or stderr encoding and errors" and the type should be changed to "feature request".
Or am I completly wrong?
44 remaining items
python-dev commentedon Sep 7, 2016
New changeset 720f0cf580e2 by Steve Dower in branch 'default':
Issue bpo-6135: Adds encoding and errors parameters to subprocess
https://hg.python.org/cpython/rev/720f0cf580e2
zooba commentedon Sep 7, 2016
I also added more tests, specifically for ansi and oem encodings on Windows and the errors argument on all platforms.
davispuh commentedon Dec 3, 2016
Still can't specify encoding for getstatusoutput and getoutput. Also it uses wrong encoding by default, most of time it will be console's codepage instead of ANSI codepage which is used now.
davispuh commentedon Dec 3, 2016
And looks like only way to specify console's codepage is with
encoding=os.device_encoding(2)which will have to be used for 99% of cases. I don't see other way...zooba commentedon Dec 3, 2016
That's true, but it only applies when the subprocess is using the same call (GetConsoleCP) to determine what encoding to use to write to something *other* than the console, which is incorrect. The defaults in this cause ought to be ACP, but it depends entirely on the target application and there is no good "99%" default.
To see this demonstrated, check the difference between these two commands (with 3.5 or earlier):
python -c "import sys; print(sys.stdin.encoding)"
python -c "import sys; print(sys.stdin.encoding)" < textfile.txt
When stdin is redirected, the default encoding is not the console encoding (unless it happens to match your ACP). And ACP is not the correct encoding for a file anyway - it just happens to be the default for TextIOWrapper - so to do it properly you need to detect that it's not a console (sys.stdin.isatty()) and then use the raw stream rather than the default wrapper. (Or you specify that your program requires ACP files, or that PYTHONIOENCODING should be set, or that it requires any other encoding and you explicitly switch to that one.)
In short, character encoding is hard, and the only way to get it right is for the developer to be aware of it. No good defaults exist.
andyclegg commentedon Oct 10, 2017
The commit for this bug (720f0cf580e2) introduces encoding and errors arguments but doesn't actually document what the values of these should be. In the case of the encoding it could be reasonably guessed, but the only way to determine what the value of 'errors' should be is to make the logical leap and look at the TextIOWrapper doc page.
And in reply to message bpo-274510, there certainly should be a 'text=True' argument added. There are countless use cases where text rather than bytes is the expected behaviour, and currently this has to be triggered by using a more obscure option. In any case, it could be argued that text should be returned when shell=True since this mimics the behaviour of any shell program.
(Sorry if adding a comment to a closed bug is poor etiquette; it seemed like the best place to put this comment)
zooba commentedon Oct 10, 2017
Do you mean it doesn't document that they take the normal encoding/errors arguments that all the other functions that do character encoding take? Or they don't specify which encoding you should use? The latter is not feasible, and if you mean the former then feel free to open a new issue for documentation.
Having a single "text" parameter is essentially what we had before - a magic option that modifies your data without giving you any ability to do it correctly. This change brought Popen in line with all the other places where we translate OS-level bytes into Python str, and it's done it in exactly the same way.
Also see msg274562 where Nick withdraws his own suggestion of a "text" parameter.
andyclegg commentedon Oct 11, 2017
I meant the former; I'll look a bit more at the documentation and submit an issue/patch.
As regards the 'text' flag - universal_newlines is actually exactly that already. I've just checked the code of subprocess.py and the universal_newlines argument is read in only two places:
So subprocess *already* has a text mode and a 'magic option' to trigger it. It works well, and is useful in most cases. When the default encoding guess is incorrect, it can easily be corrected by supplying the correct encoding. This is a good situation!
What is not so good is the API. I'm teaching a Python course for scientists at the moment. Retrieving text from external processes is an extremely common use case. I would rather not teach them to just have to use 'encoding=utf-8', because arguably teaching a user to supply an encoding without knowing if it's correct is worse than the system guessing. Equally, teaching 'universal_newlines=True' is a bit obscure.
The way forward seems to be:
ncoghlan commentedon Oct 11, 2017
This discussion should probably be submitted as a new RFE (requesting "text" as a more obvious and beginner friendly alias for universal_newlines), but I'll also add a note regarding precedent for a simple binary/text toggle: the mode settings for open().
There, the default is text (with the default encoding determined by the system), and you have to include "b" in the mode settings to say "I want raw binary".
For the subprocess APIs, the default is different (i.e. binary), so the natural counterpart to the "b" mode flag in open() would be an explicit "text" mode flag.
andyclegg commentedon Oct 11, 2017
RFE submitted as bpo-31756 , thanks
gpshead commentedon Feb 7, 2018
New changeset fc1ce81 by Gregory P. Smith (Brice Gros) in branch 'master':
bpo-6135: Fix subprocess.check_output doc to mention changes in 3.6 (GH-5564)
fc1ce81
gpshead commentedon Feb 7, 2018
New changeset 4e7a964 by Gregory P. Smith (Miss Islington (bot)) in branch '3.7':
bpo-6135: Fix subprocess.check_output doc to mention changes in 3.6 (GH-5564) (GH-5572)
4e7a964
gpshead commentedon Feb 7, 2018
New changeset 7f95c8c by Gregory P. Smith (Miss Islington (bot)) in branch '3.6':
bpo-6135: Fix subprocess.check_output doc to mention changes in 3.6 (GH-5564) (GH-5573)
7f95c8c