Repository navigation
site: Potential UnicodeDecodeError when handling pth file #87380
Description
| BPO | 43214 |
|---|---|
| Nosy | @vstinner, @methane |
| PRs | |
| Superseder |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2021-03-17.08:54:25.835>
created_at = <Date 2021-02-13.08:53:26.246>
labels = ['easy', 'type-bug', '3.8', '3.9', '3.10', 'invalid', 'library']
title = 'site: Potential UnicodeDecodeError when handling pth file'
updated_at = <Date 2021-03-17.10:30:46.450>
user = 'https://github.com/methane'bugs.python.org fields:
activity = <Date 2021-03-17.10:30:46.450>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-03-17.08:54:25.835>
closer = 'methane'
components = ['Library (Lib)']
creation = <Date 2021-02-13.08:53:26.246>
creator = 'methane'
dependencies = []
files = []
hgrepos = []
issue_num = 43214
keywords = ['patch', 'easy', '3.8regression']
message_count = 3.0
messages = ['386916', '388911', '388916']
nosy_count = 2.0
nosy_names = ['vstinner', 'methane']
pr_nums = ['24837']
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = '37364'
type = 'behavior'
url = 'https://bugs.python.org/issue43214'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']
Activity
methane commentedon Feb 13, 2021
cpython/Lib/site.py
Line 160 in 4230bd5
When default text encoding is not UTF-8 and pth file contains non-ASCII character, it will raise UnicodeDecodeError.
methane commentedon Mar 17, 2021
locale-specific encoding is not good especially for Windows.
But we used it for a long time. Changing the encoding for pth files is breaking change.
vstinner commentedon Mar 17, 2021
Since it's a Python script, the default encoding should be UTF-8, as any Python script. I guess that most pth files don't use characters outside ASCII so it's fine.
I think that distutils made a few changes to switch UTF-8 last years, so it's possible.
wkwkw commentedon Apr 11, 2024
I'm having problem with UnicodeDecodeError in this situation.
I know it can be avoided using utf-8 mode, but utf-8 mode is not working with 3.11 nor 3.12 in my environment:
I'd appreciate if pth file encoding be utf8, or someone knows a workaround to make utf-8 mode available in python 3.11 and later.
(with python 3.10 or earlier, utf-8 mode is working fine.)
I'm using new project and package manager rye that creates a pth file that contains full path of the project, which often contains non-ascii characters in some countries. So this could happen more often when rye would get more popular I guess.
vstinner commentedon Apr 11, 2024
I suggest you to open a new issue since this one is closed.