pythonw should set sys.stdout and sys.stdin to valid objects.

Bug report

Bug description:

So not exactly a bug -- but I wouldn't call it a feature request either, so I filed it as a bug.

The issue is that pythonw on Windows sets sys.stdout and sys.stdin to None.

This is problematic because while, e.g. print() will not fail, if not do anything useful, other things like warnings and logging, etc., may try to use stdout or stderr, and they will then fail.

See #107792 and #57791

So one might say: "don't do that" -- fair enough, except the user suffering from the problem may not be the one that wrote the code that's calling stderr -- it could be in a library, for instance. Library writers are often not thinking about windowed apps, and shouldn't have to do something special to accommodate them in any case.

See this note in the PyInstaller docs:

https://pyinstaller.org/en/stable/common-issues-and-pitfalls.html?highlight=os.devnull#sys-stdin-sys-stdout-and-sys-stderr-in-noconsole-windowed-applications-windows-only

In which the solution is:

import sys
import os

if sys.stdout is None:
    sys.stdout = open(os.devnull, "w")
if sys.stderr is None:
    sys.stderr = open(os.devnull, "w")

It's not unreasonable for library authors to expect that stdout and stderr to be valid.

So couldn't pythonw add that code by default? users could override it themselves, of course.

NOTE: tested recently on py3.12 -- but this has been an issue for quite some time.

NOTE2: this isn't an issue on the Mac, because its pythonw redirects to the system off-screen console.

CPython versions tested on:

3.12

Operating systems tested on:

Windows

The current behavior is documented. Maybe some applications or libraries depend on the documented behavior in order to handle the lack of standard I/O.

Wouldn't it be better if by default pythonw had stdout & stderr as os.devnull unless a flag, e.g. PYW_STD_PIPES_NONE is set?
While this would be a change from the current documented behaviour the flag would allow any applications that do depend on this to work and it would save many developers from hitting this as an error.
Maybe it would also be a nice option to add for both streams to be redirected to a logger instance with a flag such as PYW_STD_PIPES_LOG this would have saved me on a few occasions from having to re-build windowed apps to CLI so as to see what was going wrong on an end users system.

Wouldn't it be better if by default pythonw had stdout & stderr as os.devnull unless a flag, e.g. PYW_STD_PIPES_NONE is set?
While this would be a change from the current documented behaviour the flag would allow any applications that do depend on this to work and it would save many developers from hitting this as an error.

That would still put the burden on every library maintainer ever to be aware that the standard pipes are not guaranteed to be there. Granted, the chance of it breaking would be less but it would still mean that any direct sys.stderr.write() call without an appropriate if sys.stderr is not None guard is technically broken, non-portable code.

Granted, the chance of it breaking would be less but it would still mean that any direct sys.stderr.write() call without an appropriate if sys.stderr is not None guard is technically broken, non-portable code.

If sys.stdout & sys.stderr are assigned to either os.devnull or to logger objects with INFO & ERROR levels respectively then direct calls to the write methods would not fail. It is worth noting that pythonw is technically inherently non-portable. I understand that on macOS print such writes go to a hidden terminal in windowed mode so neither pipe is None there.

Thinking about it it might be simplest to assign pythonws stdout and stderr to a logger all of the time, with the default being to attach to the logging.NullHandler this could give simpler code.

I haven't looked into the details of implementing this yet but if there is general support I would be willing to give it a go.

Assuming that the os.devull approach doesn't float, shouldn't mypy be expected to catch issues with direct sys.stdxxx access? It looks like this MaybeNone is preventing it from detecting anything. (I don't use typing at all so I don't really know the rules.)

The current behavior is documented. Maybe some applications or libraries depend on the documented behavior in order to handle the lack of standard I/O.

Well, that's always the challenge with software and bugs/misfeatures.

Obligatory xkcd reference:

https://xkcd.com/1172/

But anyway-- this only applies to windowed apps on Windows-- and it's Very unlikely that anyone is using "is None" to mean anything other than "is not a valid object" -- because there are other ways to check if an app is frozen, or a GUI app, or running in windows, etc.

So if a user wants to something other than a devnull ( or equivalent) they are probably doing something else - or should be.

Note: now that I think about it -- we should probably recommend an ETAFTP approach-- catch the Exception when trying to use it, rather than check for None.

I'd be more worried about some less obvious side effect.

To give an example, PyInstaller used to do a workaround akin to sys.stdout = io.StringIO() which fixed sys.stdout.write() but broke subprocess.run() (unless its stdout parameter was set to either PIPE or DEVNULL) because the default behaviour of using the parent process's stdout involved a sys.stdout.fileno() call somewhere deep inside subprocess.

Obviously that particular error shouldn't apply to os.devnull but I'm not confident that there won't be others.

Documented behavior is not a bug. Changing such is usually a feature-request that requires strong justification and a deprecation process.

pythonw.exe is Windows specific and intended mostly for GUI apps and their installers. For instance, CPython Windows installers use pythonw.exe in the IDLE icons and context menu entries it installs. IDLE itself, as an IDE, executes user code in contexts that connect the std streams to its GUI console. (That IDLE's handling of messages from IDLE itself could/should be improved is an IDLE issue, which would not be affected by the change proposed here) What current usage problem is the justification for a non-trivial change in python.exe?

The similar issues with the 3rd-party cross-platform PyInstaller '--noconsole' command line option, which seems to have a somewhat different purpose, is not directly relevant here. Ditto for comments about macOS behavior. Maybe when the option is given, they should set the streams to devnull for users, but that is their call. Perhaps they do not because doing so is not as trivial as suggested here.

Maybe when the option is given, they should set the streams to devnull for users, but that is their call. Perhaps they do not because doing so is not as trivial as suggested here.

We (I'm a PyInstaller maintainer) don't do it because Python doesn't do it. It would be trivial for us to but we're reluctant to deviate from how regular Python behaves, even if it would save us from a lot of invalid reports (write, flush).

Note that this isn't just about "pythonw.exe", or any other GUI app that embeds the Python DLL on Windows. The process may lack standard I/O for various reasons. The standard handles could be implicitly inherited as NULL or explicitly set to NULL in the process STARTUPINFOW record. Or "python.exe" (the console app) could be spawned without a console session via the creation flag DETACHED_PROCESS, and without standard I/O explicitly set in its STARTUPINFOW record. Regarding the latter, even a non-console app such as "pythonw.exe" can be explicit spawned with valid standard I/O (e.g. pipe or nul files) set in its STARTUPINFOW record.

On Unix systems, it's trivial to spawn a Python process that lacks one or more of the standard I/O files, though it's not normal to do so. For example:

$ python -c 'import os, subprocess; os.close(0); subprocess.run(["python", "-c", "import sys; print(sys.stdin)"])'
None

@eryksun

Documented behavior is not a bug.

No one said it is. I started this issue with that very sentence. Perhaps there's a better category for this issue -- I'm happy to re-categories it (though I can't see how) -- but this doesn't really feel like it rises to "feature request" exactly either.

Changing such is usually a feature-request that requires strong justification and a deprecation process.

Sure -- how to start that process than in an issue with a discussion like this?

Sure, changing long-standing behavior could break folks' code -- and that shouldn't be done lightly. But we also could never make anything better, ever, if we were not willing to make some changes.

So every time a change is suggested we have to balance the potential benefit and the potential damage.

Finally (for the meta-conversation), maybe I'm just being pedantic or unimaginative, but I'd really appreciate it if folks could make the distinction between:

Yes, this change would be better, but it's not worth breaking folks' code over.

and

The current behavior is a good (best?) option.

Either way, nothing changes, but I think it's helpful, particularly if it ever comes up again to be clear about it -- so folks don't think they haven't been heard or understood.

On to the actual topic:

Yes, this is only about pythonw on Windows. But that doesn't mean that pythonw on MacOS (or Linux behavior) is irrelevant to the conversation -- I think it's a goal of Python to be as platform independent as possible, and at this point, folks can make, test, run, and distribute a desktop GUI app on the Mac, and then have the exact same code fail on Windows -- very annoying (this happened to me recently, hence this issue).

I presume the reason for the current pythonw behavior is that there is no default stdout and stderr on Windows for non-console apps. I also imagine that the decision to use None was made when os.devnull didn't exist. I haven't dug into the history to confirm, because it's not relevant anymore. But I'm assuming choosing None over os.devnull wasn't a carefully considered decision -- if it were, it would be great if anyone could recall why that choice was made.

But now, a few key points:

First: sys.stdout and sys.stderr are often used in libraries, rather than directly by a top level app. So:

It's reasonable for library writers to assume that sys.stdout and sys.stderr are valid objects. Do we really want everyone to have to wrap every call to them with a try: block? And (I think) that that assumption holds everywhere except

When running under pythonw or
If the streams have been reset somewhere else, which would be the fault of the higher level code that resets them.

Frankly, desktop apps on Windows are a pretty small fraction of Python use these days -- many (most?) library developers will have no idea that there is a special consideration for desktop Windows apps, and will likely never test their code under pythonw
Even if the above weren't true, it would probably be a bad idea for a library to reset these streams -- that should be left to the top-level app.

Setting them to os.devnull would address all of the above.

In practice there are two cases for desktop apps that might use pythonw:

The app author doesn't care (or hasn't thought about) where the std streams go -- in which case os.devnull would be a fine option -- and as close as we can get to what happens under Mac or Linux. (unless there IS an equivalent to the Mac's console on Windows -- in which case, let's use that! -- but I don't think there is :-( ).
The app author does care about where the std streams go -- and they write code to set them up the way they want -- redirecting to a log file, displaying in a text window, what have you. In that case, they are redirecting them anyway, it makes no difference at all what the default is.

So -- the proposed change would be better for some folks, and make no difference to others.

In short setting the streams to None does no good for anyone.

I hope I've made the case that the proposed change would have been a better way to handle this in the first place -- but it is worth potentially breaking code to make the change now?

As pointed out earlier in the thread the proposed change would break code that DOES care about the std streams, AND uses is None to detect whether to change / set it.

I suggest that that's a very small set of use-cases -- it's very hard to know for sure of course, but I'm having a hard time imagining an app author caring about where the std streams go, but ONLY if they're None if they are set to who knows what else, then they don't care?

One thing I can imagine is a developer that is only testing on Windows -- then they may use the is None approach, because they don't think carefully about it, and may not know that they will not be None under, e.g. MacOS. In which case they have accidentally written platform specific code. But even then, that would only be for an app that will be run either with a console, in which case they don't want to redirect, or without, in which case, they do -- not common?

Even for those that do use this heuristic, it could be a fairly easy to find and fix if need be. I know it's better to not introduce a breaking change at all, but in fact, this isn't all that different than the situation now. My example:

This came up for me with an app of mine that's been working fine for over 20 years.

I just updated it with some small changes, and to use a recent Python version and libraries.

First of all: updating to recent versions required a number of changes -- that shouldn't surprise anyone -- it's the cost of keeping current.

But relevant to the thread, this issue suddenly arose for me, not because of anything I'd changed, but because one of the libraries I'm using introduced a use of sys.stderr that it hadn't had before -- see above, I doubt it dawned on that library writer that they might break someone's code with this change -- but it did, due to this unfortunate default.

(and it was hard for me to detect, because I was testing on a Mac)

The point being -- making this change will introduce some bugs that need to be fixed, but it also will reduce some bugs as well.

The fact is that poorly chosen defaults cause problems -- particularly when authors of the code that uses those defaults (e.g. library authors) is different from the person that actually has to override the defaults (window'd application authors).

@eryksun wrote:

The process may lack standard I/O for various reasons.

I don't understand the implications of this. if one starts up a python app in a process without standard I/O -- will sys.stdout and sys.stderr always be set to None? or might they be set to something that will then error out when you try to use it?

Are you suggesting that they could be set to os.devnull in this case as well? I like that idea.

In short -- how does this pertain to this proposal?

... PyInstaller [doesn't] do it because Python doesn't do it.

Hmm -- unless someone does come up with a reason, "Python doesn't do it" because of legacy, and the desire not to make a breaking change.
Perhaps those some motivations don't apply to PyInstaller. In which case, I suppose it might make sense for PyInstaller to do it -- the breaking change would be in PyInstaller, and only be introduced when the user built a bundle out of an already working app that used the is None check pretty rare, and clear that it's a PyInstaller issue.

But my point is that pythonw IS a nano GUI framework

Okay, except it's not :) It's an executable with the system flag that says "don't create or attach a console" set, since there's no feasible way for a regular user to do that themselves. Code-wise it is literally identical to python.exe.

I'm not totally opposed to a PYTHON_* environment variable to force absent streams to a no-op object rather than None. But I'm not sure it's worth it. And I am opposed to a pythonw-specific environment variable, or a Windows-specific environment variable - if we can't use it everywhere, we should find another approach.

And I am opposed to a pythonw-specific environment variable, or a Windows-specific environment variable

me too -- I didn't mean to suggest that.

if it's a flag or environment variable, it should be usable everywhere.

I don't think that changing this behavior is a good idea. Too much code rely on the current assumption that standard streams are set to None if their file descriptor is set to None.

What exactly are these use cases where a piece of code would do something differently based on a sys.stdout is None check? The only case I can think of is something like...

if sys.stdout is not None:
    print("some output")
else:
    # The user can't see what we print. Better display it some other way...
    something_involving_log_files_or_pop_up_dialogs("some output")

... but that doesn't even work on Linux or macOS since a GUI application launched by the desktop still has valid <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'> streams.

A search for if sys.stdout is None in public reposirories gives a few hundred hits and every single relevant one of them is either setting sys.stdout to devnull or no-op-ing some console output function (and note that about half of them have comments indicating that this was an unpleasant surprise, and usually a reported bug). If there are other use cases then they are extremely niche.

The "something different" is only really going to be deciding to set it to something else. If it's already set, that indicates that the caller has overridden it in some way, and if not then it's up to the app to pick a reasonable default (such as devnull if it doesn't matter, or some GUI element or log file if it does).

If someone's log files disappear because we change this, they won't be very happy, and it will be our fault.

But why would someone selectively attach sys.stdout to a log file based on a conditional that so inaccurately reflects whether the current sys.stdout is going anywhere readable?

I hate to argue against my own point, but:

someone would selectively attach sys.stdout to a log file based on a conditional that so inaccurately reflects whether the current sys.stdout is going anywhere readable because that's the only obvious way to do it with the current code base.

None means "invalid stream" -- and it's pretty standard practice in Python to use that sentinel to switch on:

If there's a valid stream, the user set it up the way they want, if not, I'd better set it up to a log file.

Which is why I'm only advocating for changing it in pythonw -- in that case, it's far less likely that application writers are doing anything with "is None" than making they error go away.

All the being said -- I suspect it FAR more common for library writers to assume they will be valid, and top-level app writers to set up a redirect to logfile or whatever in a more robust way.

Being able to argue against your own point is a very good thing :) You still get to dismiss the argument.

And you're spot on. The conditional is literally "is stdout currently valid; if not, I'll make it valid", so there's nothing inaccurate about it.

Inaccurate conditionals would be "am I running under pythonw.exe? If so, sys.stdout must be invalid" or the inverse: "if sys.stdout None? If so, I must be running under pythonw.exe". So as long as we're not encouraging that line of thinking, we won't be making things worse.

I suspect it FAR more common for library writers to assume they will be valid

Yeah, and this is the core point. The choice we have to make is whether to make this assumption more likely, more obvious, or always correct.

I'm of the opinion that "more obvious" is the only option, because part of the assumption is that anything written to stdout will be seen by the user, which we can't achieve consistently when there's no console available.

Hmm, fair enough. Although I still think we'd be fixing way more problems than we'd create by making this change. Like I said before, the github search shows many many cases of people being burnt by the current model and not a single case that would lose its log file or be otherwise negatively affected by this change.

GitHub searches are a biased sample, so they only get weighed carefully (though I didn't have to go far to find one that would break). What's more important is what we've told people to do in the past: 58cb1b8

After a long discussion about the problem with Windows GUI apps Guido decided that sys.stdin, stdout and stderr should be None when the C runtime library returns invalid file descriptors for the standard streams.

I haven't tracked down the original discussion from 17 years ago, but the decision is clear enough.

So rather than just looking at code that people have published publicly in, really, the last few years, we need to weigh that for almost two decades, the behaviour has been for these to be None.

So what we need to do is figure out how to let people know that this is changing, bearing in mind that many apps are closed source, many are going to skip versions, and most already work today and shouldn't stop working just to make it so other peoples' new code appears to work first time. The challenge is the transition plan, not coming up with strong enough rationale.

In macOS Terminal: pythonw opens interactive 2.7 in Terminal, the same as python. Perhaps it symlinks to python. I am surprised that this works, and puzzled as to why. Does not matter. Neither pythonw3 nor python3w work (command not found), whereas python3 does. From a user perspective, pythonw.exe seems absent for python3. I looked up the file hierarchy from idlelib and found Python, python3, and python3.13 executables, but no 'w' versions.

@ned-deily

The difference between a command line app and a GUI app on the Mac is that a GUI app must be run from "within an application bundle".

My information is old, but back in the day, the "Framework Build" for Mac (the one distributed by python.org) had a hack to the python executable that made it appear to be inside an application bundle. It was decided that there was no point in having a separate python executable, as there was essentially no cost to the redirection.

So yes, pythonw and python are the same thing on a Framework build on the Mac.

But they are BOTH "gui apps" as opposed to both being "console apps" (which I suppose is the case on Linux, but for the opposite reason)

The core problem here is that Windows doesn't have a default console for GUI apps -- the Mac does, so on the Mac, there's nothing special to be done for the std streams for GUI apps.

@ned-deily: Is this all the same as it once was?

NOTE: the python builds for the Mac on conda-forge DO have a separate pythonw executable -- the "regular" Python is a straight *nix build, and won't work with GUI apps -- there is a separate python.app package that you need to get an executable you can run a GUI app with.

Is this all the same as it once was?

Sorry, I haven't been following this issue closely and I am not sure how relevant my reply will be because I'm not intimately familiar with the differences between python and pythonw behavior on Windows. And it sounds like there isn't a perfect analogy to a Windows console on other platforms.

AFAIK, it is true that, for python3, on non-Windows platforms built using the cpython configure and Makefile, including on macOS, we don't install any python*w executable links. The main reason for this is that, on those platforms, you normally need to be running in a platform-provided terminal window of some sort, typically but not always under a Unix shell, and typically the standard files (stdin, stdout, and stderr) are by default connected to the terminal window by the terminal emulator, in which case, python doesn't need to do anything special to interact with the user, just like any other system program run in the terminal window. (Historical note: I believe that Python running on the original classic MacOS - not macOS/OSX/Mac OS X - 25+ years ago did have need for a pythonw as the user environment there was more similar to Windows environments of the era - but that is all before my time.)

There is a separate issue of what happens when that program uses a GUI framework to create its own windows, like using Tk in a python tkinter application, including IDLE. Typically, on non-Windows platforms, IDLE can be launched from a terminal window (using with one of the idle3 commands or with a python3 -m idlelib commands) in which case the python interpreter running IDLE will normally start with stderr, stdout, and stdin connected to the initial terminal window. IDLE will redirect stdin and stdout within the Tk windows it creates but it doesn't do anything about stderr (IIRC) in the python interpreter process itself, thus, some error messages will show up in the initial terminal window.

Some Python distributions, including the python.org macOS installer, provide an additional way to launch IDLE; for macOS, there may be an IDLE.app macOS application with a clickable icon. When IDLE is launched this way, there is no terminal or Unix shell involved and thus the python interpreter's process does not have a usable stdin or stdout and stderr is directed by the OS to the system log (I believe). So some relevant messages may not be obvious to the user (see related issue #57791).

Also, on macOS, I believe that whether or not python is built as a framework build, which does create a hidden Python.app bundle to make things easier for GUI frameworks and other macOS GUI-related system calls, isn't really relevant to this discussion. stdin, stdout, and stderr should behave as described above with either a standard unix build or a framework build.

Hope that helps!

@ned-deily

Thanks @ned-deily!

Am I correct that the "python" executable in the current python.org build still have the hack that lets it run as a GUI app? i.e. it's "inside an app bundle"?

You are correct but, as I noted in my edited comment, I don't think that is relevant to this issue; it should have no impact on stdin, stdout, and stderr usage and behavior.

Thanks Ned. This means that (paraphrased) "IDLE cannot import tkinter. Bye" should go to stderr, not stdout. I should check.

Navigation Menu

Repository navigation

pythonw should set `sys.stdout` and `sys.stdin` to valid objects. #122633

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

17 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

pythonw should set sys.stdout and sys.stdin to valid objects. #122633

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Activity

eryksun commented on Aug 3, 2024

GadgetSteve commented on Aug 3, 2024

bwoodsend commented on Aug 3, 2024

GadgetSteve commented on Aug 3, 2024

bwoodsend commented on Aug 3, 2024

ChrisBarker-NOAA commented on Aug 3, 2024

bwoodsend commented on Aug 3, 2024

terryjreedy commented on Aug 3, 2024

bwoodsend commented on Aug 3, 2024

eryksun commented on Aug 3, 2024

ChrisBarker-NOAA commented on Aug 4, 2024

17 remaining items

zooba commented on Aug 6, 2024

ChrisBarker-NOAA commented on Aug 6, 2024

bwoodsend commented on Aug 6, 2024

zooba commented on Aug 6, 2024

bwoodsend commented on Aug 7, 2024

ChrisBarker-NOAA commented on Aug 7, 2024

zooba commented on Aug 7, 2024

bwoodsend commented on Aug 7, 2024

zooba commented on Aug 7, 2024

terryjreedy commented on Aug 26, 2024

ChrisBarker-NOAA commented on Aug 26, 2024

ned-deily commented on Aug 27, 2024

ChrisBarker-NOAA commented on Aug 27, 2024

ned-deily commented on Aug 27, 2024

terryjreedy commented on Aug 28, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

pythonw should set `sys.stdout` and `sys.stdin` to valid objects. #122633