IDLE 3.x on Windows exits when pasting non-BMP unicode

@terryjreedy

BPO	13153
Nosy	@terryjreedy, @vstinner, @taleinat, @ned-deily, @ezio-melotti, @aivarannamaa, @serhiy-storchaka, @animalize, @miss-islington
PRs	bpo-13153: Test IDLE Windows no-color astral paste fix #16363 bpo-13153: _tkinter part of tkinter_pythoncmd_args_2 for paste fix #16365 bpo-13153: Use OS native encoding for converting between Python and Tcl. #16545 [3.8] bpo-13153: Use OS native encoding for converting between Python and Tcl. (GH-16545) #16580 [3.7] bpo-13153: Use OS native encoding for converting between Python and Tcl. (GH-16545) #16581
Files	tkinter_nobmp_error.patch tkinter_string_conv_3.patch tkinter_pythoncmd_args.patch tkinter_pythoncmd_args_2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2019-10-04.11:44:01.512>
created_at = <Date 2011-10-11.20:01:32.297>
labels = ['type-bug', 'expert-tkinter', '3.9', '3.8', 'expert-IDLE', '3.7', 'expert-unicode']
title = 'IDLE 3.x on Windows exits when pasting non-BMP unicode'
updated_at = <Date 2019-12-20.12:13:44.282>
user = 'https://github.com/jbvsmo'

bugs.python.org fields:

activity = <Date 2019-12-20.12:13:44.282>
actor = 'terry.reedy'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2019-10-04.11:44:01.512>
closer = 'serhiy.storchaka'
components = ['IDLE', 'Tkinter', 'Unicode']
creation = <Date 2011-10-11.20:01:32.297>
creator = 'JBernardo'
dependencies = []
files = ['28376', '31610', '33318', '34016']
hgrepos = []
issue_num = 13153
keywords = ['patch']
message_count = 77.0
messages = ['145363', '145366', '145369', '145573', '145580', '145581', '145584', '145585', '145605', '145607', '145611', '145616', '145635', '155799', '155801', '155810', '155814', '155857', '177750', '177778', '177812', '179276', '182106', '182130', '182141', '182166', '182172', '182180', '182181', '182184', '182207', '182305', '182312', '192240', '192243', '192260', '193566', '194564', '194586', '194603', '194623', '196958', '196993', '201457', '207370', '207371', '207381', '210786', '254165', '254170', '318862', '319247', '348153', '352661', '353083', '353123', '353152', '353761', '353765', '353766', '353767', '353769', '353812', '353833', '353848', '353897', '353898', '353901', '353903', '353910', '353916', '353918', '353919', '353937', '353945', '358701', '358704']
nosy_count = 9.0
nosy_names = ['terry.reedy', 'vstinner', 'taleinat', 'ned.deily', 'ezio.melotti', 'Aivar.Annamaa', 'serhiy.storchaka', 'malin', 'miss-islington']
pr_nums = ['16363', '16365', '16545', '16580', '16581']
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue13153'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

I was playing with some unicode chars on Python 3.2 (x64 on Windows 7), but when pasted a char bigger than 0xFFFF, IDLE crashes without any error message.

Example (works fine):
>>> '\U000104a2'
'𐒢'

But, if I try to paste the above char, the window will instantly close.

The interpreter uses 2-bytes per char (UTF-16) and I don't know if that's causing the problem (as side note, why don't the default Windows build uses 4-bytes char?).

I can't check now with my Ubuntu install (UTF-32) if the problem persists.

This is related to bpo-12342. The problem is that Tcl/Tk 8.5 (and earlier) do not support Unicode code points outside the BMP range as in this example. So IDLE will be unable to display such characters but it should not crash either.

@ned

@ned

That looks like a bit different case. IDLE *can* print the char after you entered the '\Uxxxxxxxx' version of it.

It doesn't accept you to paste those caracters...

The current Windows build used 2-byte unicode chars because that is what Windows does. In 3.3, all builds will use a new unicode implementation that uses 1,2,or4 bytes as needed. But I suspect we will still have the paste problem unless we can somehow bypass the tk limitation.

Printing a Python string to the screen does not seem to involve conversion to a tk string. Or else tk blindly copies surrogate pairs to Windows even though it cannot create them.

In any case, true window-closing crashes (as opposed to an error traceback) are obnoxious bugs that we try to fix if possible. I verified this on my 64-bit Win 7 system. Thanks for the report. Feel free to look into the code if you can.

@terry

Just for comparison, on Python 2.7.1 (x32 on Windows 7) it's possible to paste the char (but can't use it) and a nice error is given.

>>> u'𐒢'
Unsupported characters in input

So the problem was partially solved but something might have happened with the 3.x port...

Searching on both source codes, I can see the following block was commented on Python3.2 but not on Python2.7 (Maybe someone removed someone else's bug fix?) and an assert was added.

#--- Lines 605 to 613 of PyShell.py

assert isinstance(source, str)
#                       v-- on Python2.7 it is types.UnicodeType instead
#if isinstance(source, str):
#    from idlelib import IOBinding
#    try:
#        source = source.encode(IOBinding.encoding)
#    except UnicodeError:
#        self.tkconsole.resetoutput()
#        self.write("Unsupported characters in input\n")
#        return

I uncommented those lines, removed the assert and deleted pycache for fresh bytecode but the error keeps happening.

This function runsource() is only called after the return key is pressed so the bug was introduced on another part of the program.

I'll search further but it's hard to do that without traceback of the error.

(Maybe runit() is the problem because it seems to build the line and call runsource(line))

------
PS: @terry Reedy
That looks nice to have different lengths for chars but what will be the impact on performance? Indexing will still be in constant time?

Just to complete my monologue:
Here's the traceback from running IDLE in cmd line.

C:\Python32\Lib\idlelib>python -i idle.py
Traceback (most recent call last):
  File "idle.py", line 11, in <module>
    idlelib.PyShell.main()
  File "C:\Python32\Lib\idlelib\PyShell.py", line 1429, in main
    root.mainloop()
  File "C:\Python32\Lib\tkinter\__init__.py", line 1009, in mainloop
    self.tk.mainloop(n)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1-2: invalid continuation byte

Not much meaningful but is better than nothing... Looks like some traceback is missing, and this one points to tkinter.

[Yes, indexing will still be O(1), though I personally consider that less important than most make it to be. Consistency across platforms and total time and space performance of typical apps should be the concern. There is ongoing work on improving the new implementation. Some operations already take less space and run faster.]

The traceback may very well be helpful. It implies that copying a supplemental char does not produce proper utf-8 encoded bytes. Or if it does, tkinter (or tk underneath it) does not recognize them. But then the problem should be the initial byte, not the continuation bytes, which are the same for all chars and which all have 10 for their two high order bits. See
https://secure.wikimedia.org/wikipedia/en/wiki/Utf-8
for a fuller explanation.

Line 1009 is the definition of Misc.mainloop(). I believe self.tk represents the embedded tcl interpreter, which is a black box from Python's viewpoint. Perhaps we should wrap the call with

try:
self.tk.mainloop(n)
except Exception as e:
<print error message with all info attached to e before exiting>

This should catch any miscellaneous crashes which are not otherwise caught and maybe turn the crash issues into bug reports -- the same way that running from the command line did. (It will still be good to catch what we can at error sites and give better, more specific messages.) (What I am not familiar with is how the command line interpreter might turn a tcl error into a python exception and why IDLE does not.)

When I copy '𐒢' and paste into the command line interpreter or Notepad++, I get '??'. I am guessing that ?? represent a surrogate pair and that Windows separately encodes each. The result would be 'illegal' utf-8 with an illegal continuation chars. An application can choose to decode the 'illegal' utf-8 -- or not. Python can when "errors='surrogate_escape" (or something like that) is specified. It might be possible to access the raw undecoded bytes of the clipboard with the third party pythonwin module. I do not know if there is anyway to do so with tk.

I wonder if tcl is calling back to Python for decoding and whether there was a change in the default for errors or the callback specification that would explain a change from 2.7 to 3.2.

Ezio, do you know anything about these speculations?

Thanks for the additional investigation. You don't see more in the traceback because the exception is occurring in the _tkinter C glue layer. I am able to reproduce the problem on some other platforms as well (e.g. Python 3.x on OS X with Carbon Tk 8.4). More later.

What do you mean by fixed? After deleting and remaking a pr_16545 branch, I see the same result for clipboard_get.

What is the result of new tests?

python.bat -m test -v -uall test_tk -m test_clipboard*

After remembering to recompile (sorry), the test passes and clipgoard_get returns the rocket. Very nice, thank you.

New changeset 06cb94b by Serhiy Storchaka in branch 'master':
bpo-13153: Use OS native encoding for converting between Python and Tcl. (GH-16545)
06cb94b

New changeset 6c3fbbc by Miss Islington (bot) in branch '3.7':
bpo-13153: Use OS native encoding for converting between Python and Tcl. (GH-16545)
6c3fbbc

New changeset dc19124 by Miss Islington (bot) in branch '3.8':
bpo-13153: Use OS native encoding for converting between Python and Tcl. (GH-16545)
dc19124

bpo-13153: Use OS native encoding for converting between Python and Tcl. (GH-16545)

WOW. That's huge. The issue with non-BMP characters has been fixed? Finally? The issue was haunting the bug tracker for at least 8 years!!!

Indeed, Serhiy, you've done an amazing job with this change and it will greatly benefit many people.

>> '\N{PERSONAL COMPUTER}'

freezes IDLE 3.7.6 (64-bit, downloaded from python.org) on macOS 10.15

Can it be because Tk 8.6.8 is still used there?

On Windows with 8.6.9, I see '\U0001f4bb' on 3.7.5 and '💻' on 3.8.0 and 3.9.0a0. I don't know why the difference as Serhiy's patch was backported. I will upgrade 3.7 and 3.8 and try again.

The two comments above are obsolete. I copied my comments about possible followups to a draft issue (General) on the IDLE project.

Navigation Menu

Repository navigation

IDLE 3.x on Windows exits when pasting non-BMP unicode #57362

82 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

IDLE 3.x on Windows exits when pasting non-BMP unicode #57362

Description

Activity

jbvsmo commented on Oct 11, 2011

ned-deily commented on Oct 11, 2011

jbvsmo commented on Oct 11, 2011

terryjreedy commented on Oct 15, 2011

jbvsmo commented on Oct 15, 2011

jbvsmo commented on Oct 15, 2011

terryjreedy commented on Oct 15, 2011

ned-deily commented on Oct 15, 2011

82 remaining items

terryjreedy commented on Oct 4, 2019

serhiy-storchaka commented on Oct 4, 2019

terryjreedy commented on Oct 4, 2019

serhiy-storchaka commented on Oct 4, 2019

miss-islington commented on Oct 4, 2019

miss-islington commented on Oct 4, 2019

vstinner commented on Oct 4, 2019

taleinat commented on Oct 4, 2019

aivarannamaa commented on Dec 20, 2019

terryjreedy commented on Dec 20, 2019

terryjreedy commented on Jun 2, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions