Repository navigation
python -m pdb <utf-8 encoding file.py> return UnicodeDecodeError on Windows 8.1/10 (code page 936) #103578
Description
how to replicate the issue
Python 3.11.3/3.11.2/3.11.1/3.11.0 (python-3.11.3-amd64.exe etc)
Microsoft Windows [Version 10.0.19044.2130] (Active code page: 936)
tmp.py
#coding:utf-8
print("中文")python -m pdb tmp.py
Traceback (most recent call last):
File "C:\Users\xond\AppData\Local\Programs\Python\Python311\Lib\pdb.py", line 1774, in main
pdb._run(target)
File "C:\Users\xond\AppData\Local\Programs\Python\Python311\Lib\pdb.py", line 1652, in _run
self.run(target.code)
^^^^^^^^^^^
File "C:\Users\xond\AppData\Local\Programs\Python\Python311\Lib\pdb.py", line 167, in code
return f"exec(compile({fp.read()!r}, {self!r}, 'exec'))"
^^^^^^^^^
UnicodeDecodeError: 'gbk' codec can't decode byte 0xae in position 28: illegal multibyte sequence
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> c:\users\xond\appdata\local\programs\python\python311\lib\pdb.py(167)code()
-> return f"exec(compile({fp.read()!r}, {self!r}, 'exec'))"
Activity
gaogaotiantian commentedon Apr 16, 2023
This is an encoding issue - we can simply use
io.read_codeto avoid it, as we are always expecting the read code withpdb. PR is done with the regression test.gh-103578: Fix pdb reading code with non-utf8 encoding (#103581)
pythongh-103578: Fix pdb reading code with non-utf8 encoding (pythonG…
[3.11] gh-103578: Fix pdb reading code with non-utf8 encoding (GH-103581
pythongh-103578: Fix pdb reading code with non-utf8 encoding (python#…
hauntsaninja commentedon Apr 26, 2023
Thanks again!
zooba commentedon Apr 27, 2023
This is a case that always should've gone through
io.open_code, and people may be caught out by having code execution bypass that path.We should backport to 3.8+
zooba commentedon Apr 27, 2023
I take that back - 3.8-3.10 were already fixed, it was a change in 3.11 that regressed this. So no backports required.
hauntsaninja commentedon Apr 27, 2023
Thanks for checking, sorry I could have more explicitly mentioned this! :-)