Skip to content

detect_encoding should fail with SyntaxError on invalid encoding #59195

Closed
Closed
detect_encoding should fail with SyntaxError on invalid encoding#59195
Labels
stdlibStandard Library Python modules in the Lib/ directorytopic-unicodetype-bugAn unexpected behavior, bug, or error

Description

@florentx
florentx
mannequin
opened on Jun 3, 2012
BPO 14990
Nosy @vstinner, @ezio-melotti, @florentx
PRs
  • bpo-33338: [lib2to3] Synchronize token.py and tokenize.py with stdlib #6572
    		•
    Files
  • issue14990_detect_encoding.diff
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2012-07-07.10:29:50.995>
created_at = <Date 2012-06-03.10:29:02.107>
labels = ['type-bug', 'library', 'expert-unicode']
title = 'detect_encoding should fail with SyntaxError on invalid encoding'
updated_at = <Date 2018-04-23.01:09:05.686>
user = 'https://github.com/florentx'

    bugs.python.org fields:

    activity = <Date 2018-04-23.01:09:05.686>
actor = 'lukasz.langa'
assignee = 'none'
closed = True
closed_date = <Date 2012-07-07.10:29:50.995>
closer = 'flox'
components = ['Library (Lib)', 'Unicode']
creation = <Date 2012-06-03.10:29:02.107>
creator = 'flox'
dependencies = []
files = ['25805']
hgrepos = []
issue_num = 14990
keywords = ['patch']
message_count = 7.0
messages = ['162205', '162206', '162303', '162428', '162429', '164811', '164812']
nosy_count = 4.0
nosy_names = ['vstinner', 'ezio.melotti', 'flox', 'python-dev']
pr_nums = ['6572']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue14990'
versions = ['Python 3.2', 'Python 3.3']

    Activity

    1. florentx commented on Jun 3, 2012

      @florentx
      florentxmannequin
      on Jun 3, 2012
      MannequinAuthor

      I've hit this issue while playing with tokenize for the PEP-8.py module.

      The tokenize detect_encoding() should report SyntaxError when the encoding is improperly declared.

      However it raises a LookupError in some cases.

      $ ./python -m tokenize Lib/test/bad_coding2.py 
unexpected error: unknown encoding: utf8-sig
Traceback (most recent call last):
  File "./Lib/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "./Lib/runpy.py", line 75, in _run_code
    exec(code, run_globals)
  File "./Lib/tokenize.py", line 686, in <module>
    main()
  File "./Lib/tokenize.py", line 656, in main
    tokens = list(tokenize(f.readline))
  File "./Lib/tokenize.py", line 489, in _tokenize
    line = line.decode(encoding)
LookupError: unknown encoding: utf8-sig
    2. florentx
      added
      stdlibStandard Library Python modules in the Lib/ directory
      type-bugAn unexpected behavior, bug, or error
      on Jun 3, 2012

    3. florentx commented on Jun 3, 2012

      @florentx
      florentxmannequin
      on Jun 3, 2012
      MannequinAuthor

      This patch seems to fix the issue.

    4. vstinner commented on Jun 4, 2012

      @vstinner
      vstinner
      on Jun 4, 2012
      Member

      The patch is correct according to the PEP-263:

      If a source file uses both the UTF-8 BOM mark signature and a
magic encoding comment, the only allowed encoding for the comment
is 'utf-8'.  Any other encoding will cause an error.

      The fix should also be applied to 3.2.

      (Note: Python 3.1 doesn't accept bugfixes anymore.)

    5. vstinner
      added
      topic-unicode
      on Jun 4, 2012

    6. florentx commented on Jun 6, 2012

      @florentx
      florentxmannequin
      on Jun 6, 2012
      MannequinAuthor

      It should raise a SyntaxError, if coding is 'utf8'.
      I don't agree with the last patch proposed.

      If the import report a SyntaxError, 'tokenize' should do the same.

      $ ./python Lib/test/bad_coding2.py
  File "Lib/test/bad_coding2.py", line 1
SyntaxError: encoding problem: utf-8

      and it complies strictly with PEP-263.

    7. vstinner commented on Jun 6, 2012

      @vstinner
      vstinner
      on Jun 6, 2012
      Member

      Oops, I didn't want to attach my patch to the issue. Mine is wrong, whereas yours is the right fix :-)

    8. python-dev commented on Jul 7, 2012

      @python-dev
      python-devmannequin
      on Jul 7, 2012
      Mannequin

      New changeset 5020afc0b7c9 by Florent Xicluna in branch '3.2':
      Issue bpo-14990: tokenize: correctly fail with SyntaxError on invalid encoding declaration.
      http://hg.python.org/cpython/rev/5020afc0b7c9

    9. florentx commented on Jul 7, 2012

      @florentx
      florentxmannequin
      on Jul 7, 2012
      MannequinAuthor

      Thanks. Fixed in trunk too, changeset b4322ad1fec4

    10. florentx
      closed this as completedon Jul 7, 2012
    11. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      stdlibStandard Library Python modules in the Lib/ directorytopic-unicodetype-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions