Repository navigation
untabify.py fails on files that contain non-ascii characters #53807
Description
| BPO | 9598 |
|---|---|
| Nosy | @abalkin, @pitrou, @merwok, @florentx, @PCManticore |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/abalkin'
closed_at = <Date 2010-11-30.17:32:22.716>
created_at = <Date 2010-08-13.23:06:12.165>
labels = ['type-bug', 'expert-unicode']
title = 'untabify.py fails on files that contain non-ascii characters'
updated_at = <Date 2010-11-30.17:32:22.714>
user = 'https://github.com/abalkin'bugs.python.org fields:
activity = <Date 2010-11-30.17:32:22.714>
actor = 'belopolsky'
assignee = 'belopolsky'
closed = True
closed_date = <Date 2010-11-30.17:32:22.716>
closer = 'belopolsky'
components = ['Demos and Tools', 'Unicode']
creation = <Date 2010-08-13.23:06:12.165>
creator = 'belopolsky'
dependencies = []
files = []
hgrepos = []
issue_num = 9598
keywords = []
message_count = 16.0
messages = ['113849', '114198', '114948', '115517', '115527', '115534', '115540', '115548', '115571', '115824', '115828', '115830', '115831', '115837', '115838', '122923']
nosy_count = 5.0
nosy_names = ['belopolsky', 'pitrou', 'eric.araujo', 'flox', 'Claudiu.Popa']
pr_nums = []
priority = 'low'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue9598'
versions = ['Python 3.2']
Activity
abalkin commentedon Aug 13, 2010
For example:
$ ./python.exe Tools/scripts/untabify.py Modules/_heapqmodule.c Traceback (most recent call last): ... (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf8' codec can't decode byte 0xe7 in position 173: invalid continuation byteI am not sure what relevant C standard has to say about using non-ascii characters in comments, but the checking tool should not fail with a traceback in such situation.
PCManticore commentedon Aug 18, 2010
Hello.
As it seems, untabify.py opens the file using the builtin function open, making the call error-prone when encountering non-ascii character. The proper handling should be done by using open from codecs library, specifying the encoding as argument.
e.g. codecs.open(filename, mode, 'utf-8') instead of simply open(filename, mode).
merwok commentedon Aug 25, 2010
The builtin open in 3.2 is similar to codecs.open. If you read the error message closely, you’ll see that the decoding that failed did try to use UTF-8.
The cause of the problem here is that the bytes used for the ç in François’ name are not valid UTF-8; I can fix that. This does not change the original purpose of this report: untabify should not die.
merwok commentedon Sep 3, 2010
Fixed encoding error in r84472 through r84474.
This bug should be reassessed and retitled. If untabify fails because a file has an incorrect encoding, is it really a problem in untabify? This is a developer’s tool, so getting a traceback here seems okay to me. Alexander, please close if you agree.
abalkin commentedon Sep 3, 2010
I disagree. I think we should use this opportunity to clarify preferred encoding for C language source files in python and make untabify produce meaningful diagnostic in case of encoding errors.
As a matter of policy, I see two possibilities:
Restrict C sources to 7-bit ASCII. (A pedantic reading of ANSI C standard would probably suggest even more restricted character set, but practically, I don't think 7-bit ASCII in C comments is likely to cause problems for any tools.
Require UTF-8 encoding for non-ASCII characters. Given that this is the default for python source code, it is likely that tools that are used for python development can handle UTF-8.
My vote is for #1. Display of non-ascii characters is still not universally supported and they are likely to be clobbered when diffs are copied in e-mails etc.
merwok commentedon Sep 3, 2010
I agree about the need to define the encoding for comments. My vote goes to #2, since I wouldn’t want to see names of authors/contributors mangled in the source. I would reconsider if a specification explicitly forbade that.
I repeat that the title of this bug is misleading: untabify does not fail with non-ASCII bytes, it failed because of invalid bytes.
abalkin commentedon Sep 3, 2010
This is a reason to write names in ASCII. While Latin-1 is a grey area because most of it's characters look familiar to English-speaking developers, I don't think you will easily recognize my name if I write it in Cyrillic and even if you do, chances are you would not be able to search for it. On the other hand, everyone who uses e-mail is likely to have a preferred ASCII spelling of his/her name.
merwok commentedon Sep 4, 2010
Oh, sorry, by “mangled” I meant “forced into ASCII”. I was not speaking about mojibake.
I don’t think there is an argument for Latin-1. Also, Latin-1 does not have characters but bytes, which are displayed as characters by good editors, like UTF-8 bytes are. The discussion is about ASCII versus UTF-8 in my opinion, let Latin-1 rest in peace.
Not so good example, since I’ve seen your name in the thread about Misc/ACKS sorting and could recognize it, by I get your idea :)
To search, I would use the “search for word under cursor” functionality.
Well, some languages have rules to handle constrained environments, like German who may use oe for ö or Italian E' for È, but for example in French there is no such workaround. Leaving accents out of words is a spelling error, nothing more or less. When I’m forced to change my name because of broken old tools I really feel the programmers behind the tool could do better. (I happen to have an ASCII-compatible nickname, which I prefer using to the ASCII-maimed version of my name where I can.)
I feel 2010 is very late to accept that we live in a wide world and that people should be able to just use their names with computer systems.
By the way, you still haven’t retitled this bug to address my other remark :)
florentx commentedon Sep 4, 2010
Other C files converted from latin-1 to utf-8 with r84485.
abalkin commentedon Sep 7, 2010
From IRC:
Me: UTF-8 was not strictly valid in ANSI C comments, so it is a bug in untabify to assume UTF-8 in C files.
Merwok: Works for me.
I am lowering the priority because it looks like untabify does not fail on the current code base. I'll follow up on python-dev to find out whether ASCII or UTF-8 should be enforced by untabify.
merwok commentedon Sep 8, 2010
Why would it be the job of untabify to report invalid non-ASCII characters in C files?
abalkin commentedon Sep 8, 2010
On Tue, Sep 7, 2010 at 8:08 PM, Éric Araujo <report@bugs.python.org> wrote:
..
Since untabify works by loading C code as text, it has to assume some
encoding. Failing with uncaught decode error (as it currently does
on non UTF-8 source) is not very user friendly. For example, the
diagnostic does not report the position of the offending character and
does not explain how to fix the source.
merwok commentedon Sep 8, 2010
My real question was: Shouldn’t this be a VCS hook instead of untabify’s job? (or in addition to untabify if you insist)
abalkin commentedon Sep 8, 2010
On Tue, Sep 7, 2010 at 8:31 PM, Éric Araujo <report@bugs.python.org> wrote:
..
Yes, VCS hook makes sense (and may almost eliminate the need to
handle invalid bytestreams in untabify). The hard question is still
the same, though: are non-ascii characters allowed in python C code?
My answer is "no".
merwok commentedon Sep 8, 2010
I agree with your reply (that’s what I meant with “works for me”, the question about untabify vs. hooks only occurred to me after our IRC exchange).
abalkin commentedon Nov 30, 2010
Committed revision 86893 that makes untabify.py respect encoding cookie in the files it processes. I don't think there is anything else that needs to be done here.