Repository navigation
Logging - Inconsistent behaviour when handling unicode #81292
Description
| BPO | 37111 |
|---|---|
| Nosy | @pfmoore, @vsajip, @tjguk, @methane, @zware, @zooba |
| PRs | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2019-06-17.16:43:59.796>
created_at = <Date 2019-05-31.09:57:34.129>
labels = ['type-feature', '3.9', 'OS-windows']
title = 'Logging - Inconsistent behaviour when handling unicode'
updated_at = <Date 2019-06-17.16:43:59.608>
user = 'https://bugs.python.org/jonathan-lp'bugs.python.org fields:
activity = <Date 2019-06-17.16:43:59.608>
actor = 'vinay.sajip'
assignee = 'none'
closed = True
closed_date = <Date 2019-06-17.16:43:59.796>
closer = 'vinay.sajip'
components = ['Windows']
creation = <Date 2019-05-31.09:57:34.129>
creator = 'jonathan-lp'
dependencies = []
files = ['48380']
hgrepos = []
issue_num = 37111
keywords = ['patch']
message_count = 20.0
messages = ['344053', '344059', '344060', '344081', '344106', '344110', '344113', '344164', '344198', '344203', '344204', '344208', '344212', '344456', '345319', '345413', '345430', '345432', '345523', '345877']
nosy_count = 8.0
nosy_names = ['paul.moore', 'vinay.sajip', 'tim.golden', 'methane', 'SilentGhost', 'zach.ware', 'steve.dower', 'jonathan-lp']
pr_nums = ['13932', '14008']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue37111'
versions = ['Python 3.9']
Activity
jonathan-lp commentedon May 31, 2019
Python is inconsistent in how it handles errors that have some unicode characters. It works to screen but fails to log
This works:
The following breaks:
This raises a unicode error:
UnicodeEncodeError: 'charmap' codec can't encode characters in position 11-13: character maps to <undefined>
Python 3.6.3
Given that the file created by the logger is utf-8, it's unclear why it doesn't work.
I found a workaround by using a Handler, but surely the loggers should all work the same way so that people don't get unpleasant surprises that even more painful to debug when things only break in certain logging modes?
SilentGhost commentedon May 31, 2019
I don't think this is true. Default file handler created using filename argument is opened with encoding None (meaning that it relies on system-default encoding, which is on Windows is not utf-8). And it also explains why your work-around works, provided you've opened file handler with explicit utf-8 encoding.
jonathan-lp commentedon May 31, 2019
It definitely claims to be "utf-8" in NotePad++. I've attached it if you want to double-check.
(Windows 7)
vsajip commentedon May 31, 2019
Loggers don't create files - handlers do. The file that you attached seems to be just a text file containing ASCII text. Any ASCII is also utf-8, so Notepad++'s assertion doesn't mean anything in this situation.
People have been using logging, on Windows, without problems, for years, often using utf-8 to encode their log files. Perhaps you need to read the documentation and look at examples more carefully - I'm not intending to be rude, but say that because using a handler isn't some kind of workaround, and you thinking that it was indicates that you're not sufficiently familiar with the documentation.
I'd suggest posting questions on Stack Overflow or the python-list mailing list, to establish whether there really seems to be a Python bug, before actually logging a Python issue. I appreciate that you're trying to improve Python.
jonathan-lp commentedon May 31, 2019
Thank you for your comments but this wasn't a question and I maintain this is a bug, or at least undesirable behaviour. I'd consider it a bug in my own software.
Reasoning:
Per this page: https://docs.python.org/3/howto/unicode.html
"UTF-8 is one of the most commonly used encodings, and Python often defaults to using it."
Doing something as basic as logging unicode shouldn't require knowledge of "handlers" - that's failing "simple is better than complex".
vsajip commentedon May 31, 2019
I reiterate my statement that it appears that you aren't sufficiently familiar with how logging is designed to work. Logging is more complex than print() or open() because it has to work flexibly in lots of different scenarios. Handlers are part of this, as are Formatters and Filters.
See my 2009 blog post
https://plumberjack.blogspot.com/2009/09/python-logging-101.html
for more information for why logging is designed the way it is.
If you want to configure a stream with utf-8 encoding, without explicitly using handlers, this is easily doable, as follows:
and, lo and behold, the file c:\temp\my_log.log contains
ERROR:root:จุด1
The use of the stream keyword parameter is clearly documented at
https://docs.python.org/3/library/logging.html#logging.basicConfig
Did you look at the basicConfig documentation before raising this issue?
jonathan-lp commentedon May 31, 2019
This seems like an attempt at victim blaming. But yes, I did. In fact, this is now the third time I've looked at that page - once before raising this issue, once before my previous reply, and now. I note that your example and nothing like your example is anywhere on that page. The word "encoding" doesn't appear anywhere on the page either. Sure "stream" is on there, but then you need to know about streams and make the association with logging which I apparently don't. You have to remember not everyone has your level of proficiency in the language. In fact, most Python users don't.
Lets put this another way - is there a reason NOT to have Unicode logging as the default? Clearly Unicode was important enough for Guido-et-al to decide to throw Python 2 under the bus.
I've listed the advantages of changing it, what are the disadvantages?
vsajip commentedon Jun 1, 2019
Er, no, it was a straight question about whether you'd read the documentation.
Er, no, it was stating that lots of people have successfully used logging on Windows with utf-8, as a way of justifying my opinion that there's no deficiency here that needs addressing.
Sure, and that's fine. Experienced developers learn stuff every day too. But responding to issues takes time (notice how many people will have been notified about this issue from the "nosy list" above; each would have to spend at least some time looking at it), so raising issues should be done only after exhausting other resources to make sure that the problem isn't in one's own level of knowledge or understanding, but an actual problem with the software. This is why I earlier suggested Stack Overflow and python-list as useful resources for improving one's knowledge and understanding.
I have no idea what you mean by this. As my example illustrated, it's quite easy to log Unicode in log files.
A simple Internet search for "basicConfig encoding" yields for me as the second result this Stack Overflow question
https://stackoverflow.com/questions/21402530
for which the accepted answer is a slight variation on what I suggested earlier.
jonathan-lp commentedon Jun 1, 2019
I don't see how I can be clearer. What are the reasons for NOT implementing logging to file be unicode as a default? Logging to screen is unicode as a default. What are the reasons for not wanting consistency?
Indeed, and it was from that question I got my solution in fact. The problem was the 30-60 minutes I wasted before that trying to figure out why my program was crashing and why it was only crashing *sometimes*. I'd written the logging part of the program a year ago and not really touched it since, so the logging module being a possible culprit was not even in my mind when the program crashed.
Yes, things are easy when you know it's necessary. It's the process of discovery that's an unnecessary waste of people's time. That's why I raised this and that's why I would consider this a bug in my own software. It's inconsistent, it invites problems, and it wastes peoples time.
5 remaining items
vsajip commentedon Jun 1, 2019
Different people have different ideas of what a default should be. You can pretty much guarantee that whatever a default is, someone will think it should be something else. The basicConfig functionality has been around in its present form since 2003, and people in general haven't had a problem with the current defaults. There is no case for changing a default unless there is a consensus among lots of people that a default needs changing.
pfmoore commentedon Jun 1, 2019
It is consistent. The encoding of
is consistent with the encoding of
Both use the system default encoding, which is not UTF-8. There is some discussion going on right now, as to whether it *should* be, but it isn't, and I wouldn't consider changing the behaviour of loging *without* changing the behaviour of open() to be consistent.
Logging to the console is consistent with the standard IO streams, and it was PEP-528 (https://www.python.org/dev/peps/pep-0528/) that made that change - before that, the standard IO streams, and logging to the console, used the console codepage.
So, while I agree that the behaviour takes a bit of work to understand, it's not (IMO) inconsistent, nor is it (IMO) a bug.
zooba commentedon Jun 3, 2019
I agree with Paul. Consistency with open() is the most important aspect here, and when we investigated changing that to UTF-8 it was decided to be too disruptive.
That said, I'm not opposed to adding encoding/errors to basicConfig if someone wants to do that (and the logging specialists are okay with it).
methane commentedon Jun 12, 2019
Hmm,, about encoding, I agree that default encoding of open() should be used.
If we change it, encoding of log files are changed unexpectedly after upgrading Python.
On the other hand, couldn't we use different default error handler?
"replace" or "backslashescape" seems better default error handler for the logging.
Is this change affects too many users?
vsajip commentedon Jun 12, 2019
I've left it as is using the same rationale as I guess open() has at the moment - "Errors should never pass silently. Unless explicitly silenced".
methane commentedon Jun 12, 2019
But open() is general purpose, and logging is for logging.
Note that stderr uses 'backslashreplace', even when PYTHONIOENCODING=ascii:strict is used.
And if we don't log something due to encoding error, doesn't it mean "errors may ignored completely silently by default"?
What logging does when UnicodeEncodeError is happened?
methane commentedon Jun 13, 2019
Hm, it seems logging does best job, show enough information to stderr.
But sometime, Python is embedded in web servers and there is no stderr. pythonw.exe may not have stderr too....
I still feel backslashreplace is better default error handler. But it's not strong opinion and it shouldn't block this issue. Go ahead.
zooba commentedon Jun 13, 2019
I think Vinay gets to make the call, but I also agree that some sort of error handling by default makes sense for logging. Especially now that it's (about to be) easier to override the default.
FWIW, I'd leave the default arguments as None and just coerce it as late as possible, before TextIOWrapper does it.
vsajip commentedon Jun 17, 2019
New changeset ca7b504 by Vinay Sajip in branch 'master':
bpo-37111: Add 'encoding' and 'errors' parameters to logging.basicCon… (GH-14008)
ca7b504