Tkinter binding involving Control-spacebar raises unicode error

@terryjreedy

BPO	1028
Nosy	@terryjreedy, @kbkaiser, @vstinner, @ned-deily, @bitdancer, @codeape2
Files	tkintertest.py _tkinter.c.patch PythonCmd_check_for_utf.diff just_for_explanation.patch issue1028.diff adjusts1.diff issue1028_2.diff _tkinter.c.2.patch: Convert 0xC080 to 0x00

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/kbkaiser'
closed_at = <Date 2011-05-11.21:20:52.642>
created_at = <Date 2007-08-26.20:48:31.672>
labels = ['expert-IDLE', 'type-bug', 'expert-tkinter', 'expert-unicode']
title = 'Tkinter binding involving Control-spacebar raises unicode error'
updated_at = <Date 2012-06-05.02:30:35.417>
user = 'https://github.com/kbkaiser'

bugs.python.org fields:

activity = <Date 2012-06-05.02:30:35.417>
actor = 'Mike.Perry'
assignee = 'kbk'
closed = True
closed_date = <Date 2011-05-11.21:20:52.642>
closer = 'kbk'
components = ['IDLE', 'Tkinter', 'Unicode']
creation = <Date 2007-08-26.20:48:31.672>
creator = 'kbk'
dependencies = []
files = ['8323', '8330', '12054', '12094', '14674', '14675', '14677', '21954']
hgrepos = []
issue_num = 1028
keywords = ['patch', 'needs review']
message_count = 45.0
messages = ['55311', '55312', '55313', '55321', '55325', '55330', '75999', '76032', '76034', '76035', '76041', '76043', '76164', '76165', '76173', '76178', '76179', '76180', '76701', '76846', '77209', '81060', '81989', '90665', '90707', '91405', '91408', '91420', '92678', '99441', '101508', '110080', '116773', '118400', '128441', '133494', '135707', '135788', '135795', '135806', '135807', '153250', '153252', '161718', '162321']
nosy_count = 11.0
nosy_names = ['terry.reedy', 'kbk', 'vstinner', 'ned.deily', 'r.david.murray', 'Bernt.R\xc3\xb8skar.Brenna', 'BreamoreBoy', 'jsprunck', 'mgstrein', 'python-dev', 'Mike.Perry']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue1028'
versions = ['Python 3.2', 'Python 3.3']

The control-spacebar binding is used in IDLE to
force open the completions window. It's causing
IDLE to exit with a utf8 decode error. Attached
is
a Tkinter cut-down
exhibiting the problem and a patch.

The cutdown runs ok on 2.6 but not on py3k because
the latter uses PyUnicode_FromString on all the
arguments and errs out when it encounters a
character outside the utf-8 range.

Strangely, on my system, control-spacebar is
sending a two byte
string, C0E8 via the %A parameter. Control-2
does
the same. Other keys with combinations of
modifier
keys send one byte.

Linux trader 2.6.18-ARCH #1 SMP PREEMPT Sun Nov
19
09:14:35 CET 2006 i686 Intel(R) Pentium(R) 4 CPU
2.40GHz GenuineIntel GNU/Linux

Can the problem be confirmed?

Using PyUnicode_FromUnicode on %A works because
the
unicode string is copied instead of decoded, and
that parameter is supposed to be unicode, in any
case.

The patch fixes the problem on my system but
should
be reviewed, especially whether the cast in the
call
to PyUnicode_FromUnicode is suitably cross-
platform.

Assigning to Neal since he's working a lot of
Unicode issues right now. I can check it in if I
get approval.

Heh, I see we have the same damn problem SF had: when a comment is
edited,
it doesn't re-wrap properly when submitted. You have to remove the
returns
manually after editing.

Nope, you have to make sure not to type too wide.

Well, maybe someday Tk will send a multibyte unicode
character. Update the patch.

I can confirm the problem and that your patch fixes the problem. Go
ahead and check it in. Thanks!

OK, thanks for the review! I suppose Tk is sending a bad string.
r57540

Sorry, I reverted r57540 because it caused segfault at IDLE exit. (See
bpo-4313) I reopened this issue.

I can reproduce it here with tk8.4, using tk8.5 doesn't cause this.

Here is a patch that doesn't use magic numbers :P
I didn't hit the problem described in bpo-4313 with this one, and
PythonCmd should be doing this anyway, but ideally we should move to
Tcl_CreateObjCommand.

Nudge: report on the Ubuntu bug tracker that this is still an issue with 3.2:

https://bugs.launchpad.net/bugs/517552

Tcl/Tk uses modified utf-8 internally. This includes using 0xC080, a multibyte Unicode null character, for embedded nulls that work with C's null terminated strings. Java does the same.

Note that typing Ctrl-space and Ctrl-2 are conventional ways to enter a null from the keyboard. That's the reason a null char is associated with those key combinations.

When Tcl exports Unicode, it is supposed to be strict utf-8. Until Tcl8.5, the %A (Unicode character corresponding to an event) was incorrectly leaking the modified Unicode null.

_tkinter.c.2.patch is narrowly focused: if PythonCmd raises a UnicodeDecodeError and if the string passed in an arg is 0xC080, it is replaced with the Unicode null 0x00.

New changeset 82cfbe2ddfbb by Kurt B. Kaiser in branch '3.1':
Issue bpo-1028: Tk returns invalid Unicode null in %A: UnicodeDecodeError.
http://hg.python.org/cpython/rev/82cfbe2ddfbb

I'm working on bpo-2857 which adds the "Modified UTF-8" ("utf-8-java"?) codec to Python. We can maybe use it instead of raising an error in 3.3?

r70039 3.1 forward ported > 3.2 > default. Will be in 3.2.1.

Having a modified utf-8 codec will be useful. That said, it is an error
for Tcl/Tk to expose modified utf-8 externally, and that was fixed at
some point in Tk8.5. Since Tk is no longer sending 0xC080 for the %A
char, switching codecs in _tkinter.c won't accomplish anything.

This fix was to correct a long-standing problem in IDLE using Tk8.4,
which is most easily solved by catching the leaked invalid null in
_tkinter.c.

It seems to me that, once you switch to modified utf-8 and allow the
embedded nulls, you have to make sure everything you are doing uses
the modified utf-8 encoding/decoding.

Hello,

I am still able to reproduce this issue with Python 3.2.2. It seems as if this bug was closed with a the note:

r70039 3.1 forward ported > 3.2 > default. Will be in 3.2.1.

This leads me to believe that either 3.2.2 has a regression or the patch never made it into 3.2.1.

Can anyone chime in with some more details?

Thanks,

Mike

Figured I should capture the exception. See below.

3.2.2+ (default, Jan 8 2012, 07:22:26)
[GCC 4.6.2]

Traceback (most recent call last):
  File "/usr/bin/idle3", line 5, in <module>
    main()
  File "/usr/lib/python3.2/idlelib/PyShell.py", line 1429, in main
    root.mainloop()
  File "/usr/lib/python3.2/tkinter/__init__.py", line 1012, in mainloop
    self.tk.mainloop(n)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 0: invalid start byte

On Win 7, a<cntl-space> brings up the box on all the latest releases: 2.7.3, 3.2.3, and 3.3.0a3. (These all come with recent tk 8.5.x.)

Mike, please retest with 3.2.3 and specify os and tk version and exactly what you entered if there is still a problem.

Looking good in 3.2.3! Tested on Debian Wheezy using packages
python3-tk 3.2.3-1 and idle3 3.2.3~rc1-2.

/*
* Mike Perry
* mike@cogs....com
*/

On Sun, May 27, 2012 at 4:10 PM, Terry J. Reedy <report@bugs.python.org> wrote:

Terry J. Reedy <tjreedy@udel.edu> added the comment:

On Win 7, a<cntl-space> brings up the box on all the latest releases: 2.7.3, 3.2.3, and 3.3.0a3. (These all come with recent tk 8.5.x.)

Mike, please retest with 3.2.3 and specify os and tk version and exactly what you entered if there is still a problem.

----------
nosy: +terry.reedy
versions: -Python 3.1

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue1028\>

Navigation Menu

Repository navigation

Tkinter binding involving Control-spacebar raises unicode error #45369

30 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Tkinter binding involving Control-spacebar raises unicode error #45369

Description

Activity

kbkaiser commented on Aug 26, 2007

kbkaiser commented on Aug 26, 2007

kbkaiser commented on Aug 26, 2007

kbkaiser commented on Aug 26, 2007

nnorwitz commented on Aug 26, 2007

kbkaiser commented on Aug 27, 2007

ocean-city commented on Nov 18, 2008

gpolo commented on Nov 19, 2008

gpolo commented on Nov 19, 2008

30 remaining items

bitdancer commented on Apr 11, 2011

kbkaiser commented on May 10, 2011

python-dev commented on May 11, 2011

vstinner commented on May 11, 2011

kbkaiser commented on May 11, 2011

kbkaiser commented on May 11, 2011

MikePerry commented on Feb 13, 2012

MikePerry commented on Feb 13, 2012

terryjreedy commented on May 27, 2012

MikePerry commented on Jun 5, 2012

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions