Repository navigation
IDLE 3.x on Windows exits when pasting non-BMP unicode #57362
Description
Activity
jbvsmo commented
on Oct 11, 2011 jbvsmomannequinMannequinAuthorI was playing with some unicode chars on Python 3.2 (x64 on Windows 7), but when pasted a char bigger than 0xFFFF, IDLE crashes without any error message.
Example (works fine): >>> '\U000104a2' '𐒢'But, if I try to paste the above char, the window will instantly close.
The interpreter uses 2-bytes per char (UTF-16) and I don't know if that's causing the problem (as side note, why don't the default Windows build uses 4-bytes char?).
I can't check now with my Ubuntu install (UTF-32) if the problem persists.
jbvsmo commented
on Oct 11, 2011 jbvsmomannequinMannequinAuthorThat looks like a bit different case. IDLE *can* print the char after you entered the '\Uxxxxxxxx' version of it.
It doesn't accept you to paste those caracters...
The current Windows build used 2-byte unicode chars because that is what Windows does. In 3.3, all builds will use a new unicode implementation that uses 1,2,or4 bytes as needed. But I suspect we will still have the paste problem unless we can somehow bypass the tk limitation.
Printing a Python string to the screen does not seem to involve conversion to a tk string. Or else tk blindly copies surrogate pairs to Windows even though it cannot create them.
In any case, true window-closing crashes (as opposed to an error traceback) are obnoxious bugs that we try to fix if possible. I verified this on my 64-bit Win 7 system. Thanks for the report. Feel free to look into the code if you can.
jbvsmo commented
on Oct 15, 2011 jbvsmomannequinMannequinAuthorJust for comparison, on Python 2.7.1 (x32 on Windows 7) it's possible to paste the char (but can't use it) and a nice error is given.
>>> u'𐒢' Unsupported characters in inputSo the problem was partially solved but something might have happened with the 3.x port...
Searching on both source codes, I can see the following block was commented on Python3.2 but not on Python2.7 (Maybe someone removed someone else's bug fix?) and an
assertwas added.#--- Lines 605 to 613 of PyShell.py
assert isinstance(source, str) # v-- on Python2.7 it is types.UnicodeType instead #if isinstance(source, str): # from idlelib import IOBinding # try: # source = source.encode(IOBinding.encoding) # except UnicodeError: # self.tkconsole.resetoutput() # self.write("Unsupported characters in input\n") # return
I uncommented those lines, removed the
assertand deleted pycache for fresh bytecode but the error keeps happening.This function
runsource()is only called after the return key is pressed so the bug was introduced on another part of the program.I'll search further but it's hard to do that without traceback of the error.
(Maybe
runit()is the problem because it seems to build the line and callrunsource(line))------
PS: @terry Reedy
That looks nice to have different lengths for chars but what will be the impact on performance? Indexing will still be in constant time?jbvsmo commented
on Oct 15, 2011 jbvsmomannequinMannequinAuthorJust to complete my monologue:
Here's the traceback from running IDLE in cmd line.C:\Python32\Lib\idlelib>python -i idle.py Traceback (most recent call last): File "idle.py", line 11, in <module> idlelib.PyShell.main() File "C:\Python32\Lib\idlelib\PyShell.py", line 1429, in main root.mainloop() File "C:\Python32\Lib\tkinter\__init__.py", line 1009, in mainloop self.tk.mainloop(n) UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1-2: invalid continuation byte
Not much meaningful but is better than nothing... Looks like some traceback is missing, and this one points to tkinter.
[Yes, indexing will still be O(1), though I personally consider that less important than most make it to be. Consistency across platforms and total time and space performance of typical apps should be the concern. There is ongoing work on improving the new implementation. Some operations already take less space and run faster.]
The traceback may very well be helpful. It implies that copying a supplemental char does not produce proper utf-8 encoded bytes. Or if it does, tkinter (or tk underneath it) does not recognize them. But then the problem should be the initial byte, not the continuation bytes, which are the same for all chars and which all have 10 for their two high order bits. See
https://secure.wikimedia.org/wikipedia/en/wiki/Utf-8
for a fuller explanation.Line 1009 is the definition of Misc.mainloop(). I believe self.tk represents the embedded tcl interpreter, which is a black box from Python's viewpoint. Perhaps we should wrap the call with
try:
self.tk.mainloop(n)
except Exception as e:
<print error message with all info attached to e before exiting>This should catch any miscellaneous crashes which are not otherwise caught and maybe turn the crash issues into bug reports -- the same way that running from the command line did. (It will still be good to catch what we can at error sites and give better, more specific messages.) (What I am not familiar with is how the command line interpreter might turn a tcl error into a python exception and why IDLE does not.)
When I copy '𐒢' and paste into the command line interpreter or Notepad++, I get '??'. I am guessing that ?? represent a surrogate pair and that Windows separately encodes each. The result would be 'illegal' utf-8 with an illegal continuation chars. An application can choose to decode the 'illegal' utf-8 -- or not. Python can when "errors='surrogate_escape" (or something like that) is specified. It might be possible to access the raw undecoded bytes of the clipboard with the third party pythonwin module. I do not know if there is anyway to do so with tk.
I wonder if tcl is calling back to Python for decoding and whether there was a change in the default for errors or the callback specification that would explain a change from 2.7 to 3.2.
Ezio, do you know anything about these speculations?
Thanks for the additional investigation. You don't see more in the traceback because the exception is occurring in the _tkinter C glue layer. I am able to reproduce the problem on some other platforms as well (e.g. Python 3.x on OS X with Carbon Tk 8.4). More later.
82 remaining items
What do you mean by fixed? After deleting and remaking a pr_16545 branch, I see the same result for clipboard_get.
What is the result of new tests?
python.bat -m test -v -uall test_tk -m test_clipboard*After remembering to recompile (sorry), the test passes and clipgoard_get returns the rocket. Very nice, thank you.
Indeed, Serhiy, you've done an amazing job with this change and it will greatly benefit many people.
aivarannamaa commented
on Dec 20, 2019 aivarannamaamannequinMannequin>> '\N{PERSONAL COMPUTER}'
freezes IDLE 3.7.6 (64-bit, downloaded from python.org) on macOS 10.15
Can it be because Tk 8.6.8 is still used there?
On Windows with 8.6.9, I see '\U0001f4bb' on 3.7.5 and '💻' on 3.8.0 and 3.9.0a0. I don't know why the difference as Serhiy's patch was backported. I will upgrade 3.7 and 3.8 and try again.
The two comments above are obsolete. I copied my comments about possible followups to a draft issue (General) on the IDLE project.
Metadata
Metadata
Assignees
Projects
- Status
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields: