Repository navigation
add an optional "default" argument to tokenize.detect_encoding #53980
Description
| BPO | 9771 |
|---|---|
| Nosy | @vstinner, @benjaminp, @merwok, @florentx, @berkerpeksag, @serhiy-storchaka |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2010-09-04.11:32:08.849>
labels = ['type-feature', 'library']
title = 'add an optional "default" argument to tokenize.detect_encoding'
updated_at = <Date 2014-11-02.12:11:24.713>
user = 'https://github.com/florentx'bugs.python.org fields:
activity = <Date 2014-11-02.12:11:24.713>
actor = 'berker.peksag'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2010-09-04.11:32:08.849>
creator = 'flox'
dependencies = []
files = ['18745']
hgrepos = []
issue_num = 9771
keywords = ['patch']
message_count = 3.0
messages = ['115567', '122106', '173002']
nosy_count = 6.0
nosy_names = ['vstinner', 'benjamin.peterson', 'eric.araujo', 'flox', 'berker.peksag', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue9771'
versions = ['Python 3.5']
Activity
florentx commentedon Sep 4, 2010
The function tokenize.detect_encoding() detects the encoding either in the coding cookie or in the BOM. If no encoding is found, it returns 'utf-8':
When result is 'utf-8', there's no (easy) way to know if the encoding was really detected in the file, or if it falls back to the default value.
Cases (with utf-8):
The proposal is to allow to call the function with a different default value (None or ''), in order to know if the encoding is really detected.
For example, this function could be used by the Tools/scripts/findnocoding.py script.
Patch attached.
vstinner commentedon Nov 22, 2010
I never understood the usage of the second item. IMO it should be None if no cookie found.
serhiy-storchaka commentedon Oct 15, 2012
UTF-8 is the default source encoding for Python 3.
serhiy-storchaka commentedon May 22, 2022
Is there any use case besides the Tools/scripts/findnocoding.py script? It is a special tool and it needs to look under the hood.
There are many ways of working with Python sources:
tokenize.open().tokenize.detect_encoding()and manually decode the rest of the file. It is the least convenient way.vstinner commentedon May 23, 2022
I suggest to close this issue, it seems like there is no real need for this feature.
AA-Turner commentedon May 28, 2022
Closing per Victor, it's been several days with no comments to the contrary.
A