Repository navigation
Unicode escape sequences not parsed in raw strings. #46793
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/benjaminp'
closed_at = <Date 2008-04-28.21:05:55.867>
created_at = <Date 2008-04-03.04:19:03.157>
labels = ['type-bug', 'docs']
title = 'Unicode escape sequences not parsed in raw strings.'
updated_at = <Date 2008-04-28.21:05:55.865>
user = 'https://github.com/jmillikin'bugs.python.org fields:
activity = <Date 2008-04-28.21:05:55.865>
actor = 'benjamin.peterson'
assignee = 'benjamin.peterson'
closed = True
closed_date = <Date 2008-04-28.21:05:55.867>
closer = 'benjamin.peterson'
components = ['Documentation']
creation = <Date 2008-04-03.04:19:03.157>
creator = 'jmillikin'
dependencies = []
files = ['9947', '9948', '9952']
hgrepos = []
issue_num = 2541
keywords = ['patch']
message_count = 24.0
messages = ['64890', '64896', '64897', '64898', '64900', '64978', '64982', '64984', '64985', '64986', '64990', '64997', '65009', '65083', '65085', '65211', '65212', '65223', '65225', '65234', '65502', '65512', '65930', '65934']
nosy_count = 7.0
nosy_names = ['lemburg', 'gvanrossum', 'nnorwitz', 'georg.brandl', 'amaury.forgeotdarc', 'benjamin.peterson', 'jmillikin']
pr_nums = []
priority = 'critical'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue2541'
versions = ['Python 3.0']
Activity
jmillikin commentedon Apr 3, 2008
According to
<http://docs.python.org/dev/3.0/reference/lexical_analysis.html#id9\>,
raw strings with \u and \U escape sequences should have these sequences
parsed as usual. However, they are currently escaped.
benjaminp commentedon Apr 3, 2008
You use the "ur" string mode.
amauryfa commentedon Apr 3, 2008
No, it's about python 3.0. I confirm the problem, and propose a patch:
benjaminp commentedon Apr 3, 2008
Thanks for noticing, Amaury, and your patch works for me.
benjaminp commentedon Apr 3, 2008
Fixed in r62128.
benjaminp commentedon Apr 5, 2008
Sorry, Guido said this is not allowed:
http://mail.python.org/pipermail/python-3000/2008-April/012952.html. I
reverted it in r62165.
gvanrossum commentedon Apr 5, 2008
The docs still need to be updated! An entry in what's new in 3.0 should
also be added.
9 remaining items
gvanrossum commentedon Apr 7, 2008
To be honest, I don't know what the uses are for that codec.
amauryfa commentedon Apr 7, 2008
pickle still uses it when protocol=0 (and cPickle as well, but in trunk/
only of course)
malemburg commentedon Apr 8, 2008
You can't change the codec - it's being used in other places as well,
e.g. for use cases where you need to have an 8-bit encoded readable
version of a Unicode object (which happens to be Latin-1 + Unicode
escapes for all non-Latin-1 characters, due to Unicode being a superset
of Latin-1).
Adding a new codec would be fine, though I don't know how this would map
raw Unicode strings with non-Latin-1 characters in them to an 8-bit
string. Perhaps this is not needed at all in Py3k.
amauryfa commentedon Apr 8, 2008
Isn't "unicode-escape" enough for this purpose?
malemburg commentedon Apr 8, 2008
What do you mean with "enough" ?
The "raw-unicode-escape" codec is used in Python 2.x to convert literal
strings of the form ur"" to Unicode objects. It's a variant of the
"unicode-escape" codec.
The codec is also being used in cPickle, pickle, variants of pickle,
Python code generators, etc.
It serves its purpose, just like "unicode-escape" and all the other
codecs in Python.
amauryfa commentedon Apr 8, 2008
I mean: now that raw strings cannot represent all unicode points (or
more precisely, they need the file encoding to do so), is there a use
case for "raw-unicode-escape" that cannot be filled by the
unicode-escape codec?
Note that pickle does not use "raw-unicode-escape" as is: it replaces
backslashes by \u005c. This has the nice effect that pickled strings can
also be decoded by "unicode-escape".
That's why I propose to completely remove raw-unicode-escape, and use
unicode-escape instead.
malemburg commentedon Apr 9, 2008
While that's true for cPickle, it is not for pickle. The pickle protocol
itself is defined in terms of the "raw-unicode-escape" codec (see
pickle.py).
Besides, you cannot assume that the Python interpreter itself is the
only use-case for these codecs. The "raw-unicode-escape" codec is well
usable for other purposes where you need a compact way of encoding
Unicode, especially if you're strings are mostly Latin-1 and only
include non-UCS2 code points every now and then. That's also the reason
why pickle uses it.
nnorwitz commentedon Apr 15, 2008
What is the status of this bug? AFAICT, the code is now correct. Have
the doc changes been applied? The resolution on this report should be
updated too. It's currently rejected.
benjaminp commentedon Apr 15, 2008
It's rejected because the OP wanted unicode escapes to be applied in
unicode strings, and I haven't applied the docs because nobody has told
me I should.
birkenfeld commentedon Apr 28, 2008
Please apply the patch, but rename "Unicode escapes" to "\u and \U
escapes" first.
benjaminp commentedon Apr 28, 2008
Fixed in r62568.