Repository navigation
parser: store the filename as an unicode object #54994
Description
| BPO | 10785 |
|---|---|
| Nosy | @abalkin, @vstinner, @benjaminp |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2011-04-04.23:56:24.097>
created_at = <Date 2010-12-28.02:40:20.736>
labels = ['interpreter-core', 'expert-unicode']
title = 'parser: store the filename as an unicode object'
updated_at = <Date 2011-04-04.23:56:24.097>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2011-04-04.23:56:24.097>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2011-04-04.23:56:24.097>
closer = 'vstinner'
components = ['Interpreter Core', 'Unicode']
creation = <Date 2010-12-28.02:40:20.736>
creator = 'vstinner'
dependencies = []
files = ['20271']
hgrepos = []
issue_num = 10785
keywords = ['patch']
message_count = 9.0
messages = ['124755', '124823', '124826', '124827', '124828', '125302', '125409', '130937', '132990']
nosy_count = 4.0
nosy_names = ['belopolsky', 'vstinner', 'benjamin.peterson', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue10785'
versions = ['Python 3.3']
Activity
vstinner commentedon Dec 28, 2010
The Python parser stores the filename as a byte string. But it decodes the filename on error because most Python functions now use unicode strings. Instead of decoding the filename at error, which may raise a new error, I propose to decode the filename on the creation of the parser object and only store the filename as unicode.
This issue would prepare the last part of the full unicode support (bpo-3080).
abalkin commentedon Dec 28, 2010
I like the idea, but I don't like the trend that parser code continues to diverge from pgen. I understand that most of the Python runtime is not available to pgen, but maybe a more elegant solution than changing the type conditional on PGEN can be found. For example, maybe filename could be decoded from FS encoding to UTF-8?
vstinner commentedon Dec 28, 2010
In pgen, the filename is only used to display the following warning, in indenterror():
<filename>: inconsistent use of tabs and spaces in indentation
In pratical, this warning never occurs on Grammar/Grammar: this file doesn't use indentation at all, only continuation lines.
A better solution is maybe just to drop the filename for pgen. Anyway, pgen only compiles *one* file (Grammar/Grammar), so we don't need the input filename ;-)
vstinner commentedon Dec 28, 2010
When testing my patch, I found and fixed two bugs in pgen:
vstinner commentedon Dec 28, 2010
Version 2 of the patch:
vstinner commentedon Jan 4, 2011
err_clear() should set err->filename to NULL.
vstinner commentedon Jan 5, 2011
Version 3 of the patch to fix also bpo-9319.
vstinner commentedon Mar 15, 2011
@benjamin: You told me that you don't want two versions of pgen, but I don't remember why. As my work on bpo-3080 is mostly done, I now plan to patch the Python parser to store the filename as Unicode. So could you please review the patch attached to this issue?
python-dev commentedon Apr 4, 2011
New changeset 6e9dc970ac0e by Victor Stinner in branch 'default':
Issue bpo-10785: Store the filename as Unicode in the Python parser.
http://hg.python.org/cpython/rev/6e9dc970ac0e