Repository navigation
[Py3k] SyntaxError cursor shifted if multibyte character is in line. #46635
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2008-03-18.05:22:30.823>
labels = ['interpreter-core', 'type-bug']
title = '[Py3k] SyntaxError cursor shifted if multibyte character is in line.'
updated_at = <Date 2018-08-18.21:23:35.490>
user = 'https://bugs.python.org/ocean-city'bugs.python.org fields:
activity = <Date 2018-08-18.21:23:35.490>
actor = 'terry.reedy'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2008-03-18.05:22:30.823>
creator = 'ocean-city'
dependencies = []
files = ['11548', '11670', '11707', '13354', '13356', '13357', '13358', '13359', '30534', '31874']
hgrepos = []
issue_num = 2382
keywords = ['patch']
message_count = 29.0
messages = ['63895', '63904', '64156', '73539', '74106', '74114', '74119', '74129', '74148', '74149', '74361', '74362', '83636', '83659', '83700', '83712', '83714', '140377', '149960', '172500', '172525', '190933', '190934', '198419', '208115', '208697', '228027', '228034', '323734']
nosy_count = 11.0
nosy_names = ['terry.reedy', 'amaury.forgeotdarc', 'belopolsky', 'vstinner', 'ocean-city', 'LambertDW', 'ezio.melotti', 'Arfrever', 'python-dev', 'petri.lehtinen', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue2382'
versions = ['Python 3.3', 'Python 3.4']
Activity
ocean-city commentedon Mar 18, 2008
Hello. I found another problem related to bpo-2301.
SyntaxError cursor "^" is shifted when multibyte
characters are in line (before "^").
I think this is because err->text is stored as UTF-8
which requires 3 bytes for multibyte character,
but actually cp932 (my console encoding) requires only 2 bytes for it.
So "^" is shited to right 5 bytes because there is 5 multibyte chars.
Sorry, I didn't know what PyTokenizer_RestoreEncoding really doing.
That function adjusted err_ret->offset for this encoding conversion.
So, Python2.5 can output cursor in right place. (Of course, if source
encoding is not compatible for console encoding, broken string is printed
though. Anyway, cursor is right)
I tried to fix this problem, but I'm not sure how to fix this.
ocean-city commentedon Mar 18, 2008
Quick observation...
///////////////////////////////////
// Possible Solution
Convert err->text to console compatible encoding (not to source
encoding like in python2.x) where PyTokenizer_RestoreEncoding is there.
err->text is UTF-8, actual output is done in
Python/pythonrun.c(print_error_text), so adjust offset there.
///////////////////////////////////
// Solution requires...
1.
PyUnicode_DecodeUTF8 in Python/pythonrun.c(err_input) should
be changed to some kind of "bytes" API.
The way to write "bytes" to File object directly is needed.
////////////////////////////////////////////////////
// Experimental patch
Attached as experimental patch of solution 2. Looks agly, but
seems working on my environment.
(I assumed get_length_in_bytes(f, " ", 1) == 1 but I'm not sure
this is always true in other platforms. Probably nicer and more
general solution may exist)
ocean-city commentedon Mar 20, 2008
This assumption still lives, but I cannot find better solution.
I'm thinking now attached patch is good enough.
ocean-city commentedon Sep 21, 2008
Patch revised.
amauryfa commentedon Sep 30, 2008
I think that your patch works only for terminals where one byte of the
encoded text is displayed as one character on the terminal. This is not
true for utf-8 terminals, for example.
In the attached patch, I tried to write some unit tests, (I had to adapt
the traceback module as well), and one test still fails because the
captured stderr has a utf-8 encoding.
I think that it's better to count unicode characters.
ocean-city commentedon Oct 1, 2008
You are right, this issue is more difficult than I thought...
I found wcswidth(3), if this function is available we can use this
function, but unfortunately there is no such function in VC6 and this
function is meaningless on cygwn, so I cannot test it. ;-(
Maybe we can use
import unicodedata
unicodedata.east_asian_width()
but I need to investigate more.
amauryfa commentedon Oct 1, 2008
For the moment, I'd suggest that one unicode character has a the same
with as the space character, assuming that stdout.encoding correctly
matches the terminal.
Then the C implementation could do something similar to the statements I
added in traceback.py:
offset = len(line.encode('utf-8')[:offset].decode('utf-8'))
ocean-city commentedon Oct 1, 2008
Amaury, if doing so, the cursor will shift left by 5 columns on my
environment like this, no? ("あ" requires 2 columns for example)
amauryfa commentedon Oct 1, 2008
This seems to be a difficult problem. Doesn't the exact width depend on
the terminal capabilities? and fonts, and combining diacritics...
An easy way to put the caret at the same exact position is to repeat the
beginning of the line up to the offending offset:
print "あいうえお"
print "あいうえお^<------------------
But I don't know how to make it look less ugly.
At least my "one unicode char is one space" suggestion corrects the case
of Western languages, and all messages with single-width characters.
vstinner commentedon Oct 2, 2008
See also a related issue: bpo-3975.
ocean-city commentedon Oct 6, 2008
I'm not happy with this solution. ;-(
I have to admit you are right.
Nevertheless, I got coLinux(Debian) which has localed wcswidth(3), so I
created another experimental patch.
(py3k_adjust_cursor_at_syntax_error_v2.patch)
The strategy is ...
(Amaury's suggestion)
This patch ignores file encoding. Again, this patch is experimental,
best effort, but maybe better than current state.
ocean-city commentedon Oct 6, 2008
Experimental patch was experimental, wcswidth(3) returns 1 for East
Asian Ambiguous character.
should point 'c'. And another one
Please forget my patch. :-(
vstinner commentedon Mar 15, 2009
This issue is a problem of units. The error text is an utf8 *byte*
string and offset is a number of *bytes*. The goal is to get the text
*width* of a *character* string. We have to:
1- convert offset from bytes number to character number
2- get the error message as (unicode) characters
3- get the width of text[:offset]
It's already possible to get (2) from the utf8 string, and code from
ocean-city's patch (py3k_adjust_cursor_at_syntax_error_v2.patch) can
be used for (3). The most difficult point is (1).
I will try to implement that.
lambertdw commentedon Mar 16, 2009
Resolution of this may be applicable to bpo-3446 as well.
"center, ljust and rjust are inconsistent with unicode parameters"
12 remaining items
python-dev commentedon Jan 21, 2014
New changeset eb7565c212f1 by Serhiy Storchaka in branch '3.3':
Issue bpo-2382: SyntaxError cursor "^" now is written at correct position in most
http://hg.python.org/cpython/rev/eb7565c212f1
New changeset ea34b2b0b8ae by Serhiy Storchaka in branch 'default':
Issue bpo-2382: SyntaxError cursor "^" now is written at correct position in most
http://hg.python.org/cpython/rev/ea34b2b0b8ae
vstinner commentedon Sep 30, 2014
The issue bpo-10384 has been marked as a duplicate of this issue: it's a similar issue, identifier which contains invisible character.
abalkin commentedon Sep 30, 2014
The original problem is still present
Replace A_I_U_E_O above with the Japanese script. I get codec error from the server when I try to paste my session as is.
(Note that invalid character is $ above and not the Japanese AIUEO.)
Another outstanding issue is with zero-width characters. See bpo-10384.
terryjreedy commentedon Aug 18, 2018
IDLE avoids the problem of calculating a location for a '^' below the bad line by instead asking tk to give the marked character (and maybe more) a 'ERROR' tag, which shows as a red background. So it marks the '$' of 'A_I_U_E_O$' and the 'alid' slice of 'inv\u200balid' (from duplicate bpo-10384). When the marked character is '\n', the space following the line is tagged. Is it possible to do something similar with any of the major system consoles?
iritkatriel commentedon Sep 28, 2022
I think it has been fixed by now. On main (3.12) I get:
ezio-melotti commentedon Nov 11, 2022
I tried the examples in the comments above and the output look correct to me.
Note however that there are two different issues. The first is about multibyte characters that have regular width (like
ä) -- and that issue is fixed:The second issue affects characters that, when displayed, are wider than normal (regardless of the number of bytes used to represent them):
For these, the cursor is preceded by the correct amount of spaces, but since the characters are wider than usual (even with a monospace font), the output looks misaligned.
IIUC this report is about the first issue, which is now fixed, so we can close this.
I'm not sure if the second issue is something we can/should fix on Python's side.
serhiy-storchaka commentedon Nov 12, 2022
The original error is still present.
This issue was not closed in 2014, because it consists of two parts, and my patch only fixed the first part. It now mostly works for many European languages (unless you use combining characters), but not for the CJK languages with double width characters and not for languages with a wide use of zero width characters.
ezio-melotti commentedon Nov 13, 2022
If you are referring to the fact that
print "あいうえお"is longer than^^^^^^^^^^^^^, is there even any way to fix this on our end?In the example you posted, there are 13 characters and thus 13
^in the error message:The alignment doesn't match simply because -- even with a monospace font -- the Hiragana character are wider than the ASCII characters.
If the
^s are simply supposed to match the number of characters, then this is no different from:If instead we want to use a number of
^s that aligns nicely with the Hiragana characters, then it's more complicated.If the Hiragana characters were twice as wide we could in theory duplicate the number of
^(even though it might be confusing to have two^^for a single character), but they aren't, so we can't do that:I also suspect the exact output depends both on the available (and used) fonts, and by the rendering system (that might e.g. decide to increase the width of the ASCII characters to match the one of the widest character in the string).
So in order to produce the "right" amount of
^s, Python will have to know the width of the string and divide it by the width of a single^. In my browser, the example above would need about 16.5^.For character that actually are twice as wide or zero-width character we might in theory duplicate or remove the
^for each such character (assuming there is a Unicode property to identify them), and the output will look right. Currently with zero-width characters it looks like:Arguably having the number of
^matching the number of characters -- in addition to being much easier to handle -- is also better in some situation. In this example it hints to the fact that there are 4 extra invisible characters.arhadthedev commentedon May 11, 2023
Superseded by and fixed in gh-102310.