typo in a module describes utf-8 as uft-8

@malemburg

BPO	4540
Nosy	@malemburg, @amauryfa
Superseder	bpo-5453: pydoc -k fails (release30-maint patch)
Files	python3bug.png: screenshot help_badencoding.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/amauryfa'
closed_at = <Date 2009-07-15.21:43:10.960>
created_at = <Date 2008-12-04.23:38:42.944>
labels = ['type-crash']
title = 'typo in a module describes utf-8 as uft-8'
updated_at = <Date 2009-07-15.21:43:10.959>
user = 'https://bugs.python.org/johnweldon'

bugs.python.org fields:

activity = <Date 2009-07-15.21:43:10.959>
actor = 'amaury.forgeotdarc'
assignee = 'amaury.forgeotdarc'
closed = True
closed_date = <Date 2009-07-15.21:43:10.960>
closer = 'amaury.forgeotdarc'
components = []
creation = <Date 2008-12-04.23:38:42.944>
creator = 'john.weldon'
dependencies = []
files = ['12229', '12239']
hgrepos = []
issue_num = 4540
keywords = ['patch', 'needs review']
message_count = 13.0
messages = ['76945', '76955', '76996', '76999', '77000', '77001', '77002', '77009', '77065', '83421', '83431', '87608', '90545']
nosy_count = 6.0
nosy_names = ['lemburg', 'amaury.forgeotdarc', 'john.weldon', 'dariusp', 'jdwhitley', 'djinn']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'patch review'
status = 'closed'
superseder = '5453'
type = 'crash'
url = 'https://bugs.python.org/issue4540'
versions = ['Python 3.0']

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "c:\Python30\lib\site.py", line 427, in __call__
    return pydoc.help(*args, **kwds)
  File "c:\Python30\lib\pydoc.py", line 1675, in __call__
    self.interact()
  File "c:\Python30\lib\pydoc.py", line 1693, in interact
    self.help(request)
  File "c:\Python30\lib\pydoc.py", line 1711, in help
    self.listmodules(request.split()[1])
  File "c:\Python30\lib\pydoc.py", line 1799, in listmodules
    apropos(key)
  File "c:\Python30\lib\pydoc.py", line 1913, in apropos
    ModuleScanner().run(callback, key, onerror=onerror)
  File "c:\Python30\lib\pydoc.py", line 1875, in run
    source = loader.get_source(modname)
  File "c:\Python30\lib\pkgutil.py", line 293, in get_source
    self.source = self.file.read()
  File "c:\Python30\lib\io.py", line 1720, in read
    decoder = self._decoder or self._get_decoder()
  File "c:\Python30\lib\io.py", line 1506, in _get_decoder
    make_decoder = codecs.getincrementaldecoder(self._encoding)
  File "c:\Python30\lib\codecs.py", line 960, in getincrementaldecoder
    decoder = lookup(encoding).incrementaldecoder
LookupError: unknown encoding: uft-8

When enumerating all possible modules, the help system tries to import 
test.bad_coding.py :-(
I tried to catch this LookupError, but then it fail on test.badsyntax_pep3120.py, 
with a SyntaxError
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python30\lib\site.py", line 427, in __call__
    return pydoc.help(*args, **kwds)
  File "c:\python30\lib\pydoc.py", line 1675, in __call__
    self.interact()
  File "c:\python30\lib\pydoc.py", line 1693, in interact
    self.help(request)
  File "c:\python30\lib\pydoc.py", line 1711, in help
    self.listmodules(request.split()[1])
  File "c:\python30\lib\pydoc.py", line 1799, in listmodules
    apropos(key)
  File "c:\python30\lib\pydoc.py", line 1913, in apropos
    ModuleScanner().run(callback, key, onerror=onerror)
  File "c:\python30\lib\pydoc.py", line 1872, in run
    loader = importer.find_module(modname)
  File "C:\Python30\lib\pkgutil.py", line 186, in find_module
    file, filename, etc = imp.find_module(subname, path)
SyntaxError: Non-UTF-8 code starting with '\xf6' in file (null) on line 1, but n
o encoding declared; see http://python.org/dev/peps/pep-0263/ for details

[Note: I don't like the (null) as a filename...]

Amaury, I don't see where imp.find_module() tries to do a compile()
(which would trigger the SyntaxError). Is that really the full traceback ?

Regarding the strategy of pydoc to try to list all modules: I don't
think that's such a good idea. It's likely to take a long time to
compile such a list on a moderately used Python system and can have lots
of unwanted side effects via import hooks.

Instead, pydoc should just say that a module was not specified and be
done with it.

Yes, it is the full traceback.

The modules are not compiled nor imported.
But to load the source code, importer.find_module() needs to open the
file in text mode, so it calls PyTokenizer_FindEncoding() (this is the
second failure, with test.badsyntax_pep3120.py), then the encoding is
used to decode text (this is the first failure, with test.bad_coding.py)

Instead, pydoc should just say that a module was not specified
and be done with it.
I'm not sure to understand.

On 2008-12-05 11:24, Amaury Forgeot d'Arc wrote:

Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:

Yes, it is the full traceback.

The modules are not compiled nor imported.
But to load the source code, importer.find_module() needs to open the
file in text mode, so it calls PyTokenizer_FindEncoding() (this is the
second failure, with test.badsyntax_pep3120.py), then the encoding is
used to decode text (this is the first failure, with test.bad_coding.py)

I see.

> Instead, pydoc should just say that a module was not specified
> and be done with it.

I'm not sure to understand.

Sorry, I didn't see the attached PNG screenshot until now. Forget that
comment.

What I don't understand is that this works:

help> modules

Please wait a moment while I gather a list of all available modules...
...

while this doesn't:

help> modules web

Here is a list of matching modules. Enter any module name to get more help.

Traceback (most recent call last):
...

Shouldn't the first method also cause a traceback ?

Explanation is simple:
help("modules") just lists all files but do not open them.
help("modules web") has to search text inside.

On 2008-12-05 11:49, Amaury Forgeot d'Arc wrote:

Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:

Explanation is simple:
help("modules") just lists all files but do not open them.
help("modules web") has to search text inside.

Ok, thanks.

What if we skip any errors that occur during this scan ? After all,
it's just a help function, not a vital system operation.

Attached patch ignores any Exception during module import.
Note that KeyboardInterrupt is not caught: it is not an Exception.

I could provide a similar patch for 2.7, if there is some interest; but
no crash was reported.

On 2008-12-05 13:28, Amaury Forgeot d'Arc wrote:

Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:

Attached patch ignores any Exception during module import.
Note that KeyboardInterrupt is not caught: it is not an Exception.

Is there a reason why "formatter" is imported inside the method
rather than at module scope ?

Other than that the patch looks good.

I could provide a similar patch for 2.7, if there is some interest; but
no crash was reported.

Python 2.6 and 2.7 have the same problem:

help> modules web

Here is a list of matching modules. Enter any module name to get more help.

...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/python-2.6-ucs2/lib/python2.6/site.py", line 429, in __call__
    return pydoc.help(*args, **kwds)
  File "/usr/local/python-2.6-ucs2/lib/python2.6/pydoc.py", line 1687, in __call__
    self.interact()
  File "/usr/local/python-2.6-ucs2/lib/python2.6/pydoc.py", line 1705, in interact
    self.help(request)
  File "/usr/local/python-2.6-ucs2/lib/python2.6/pydoc.py", line 1723, in help
    self.listmodules(split(request)[1])
  File "/usr/local/python-2.6-ucs2/lib/python2.6/pydoc.py", line 1812, in
listmodules
    apropos(key)
  File "/usr/local/python-2.6-ucs2/lib/python2.6/pydoc.py", line 1912, in apropos
    ModuleScanner().run(callback, key)
  File "/usr/local/python-2.6-ucs2/lib/python2.6/pydoc.py", line 1877, in run
    for importer, modname, ispkg in pkgutil.walk_packages(onerror=onerror):
  File "/usr/local/python-2.6-ucs2/lib/python2.6/pkgutil.py", line 110, in
walk_packages
    __import__(name)
  File "/home/lemburg/lib/DNS/__init__.py", line 13, in <module>
    import Type,Opcode,Status,Class
  File "/home/lemburg/lib/DNS/Type.py", line 67
SyntaxError: Non-ASCII character '\xf6' in file /home/lemburg/lib/DNS/Type.py on
line 67, but no encoding declared; see http://www.python.org/peps/pep-0263.html
for details

(pydoc will look in all modules on sys.path, so chances are high
that users will run into similar problems)

I can still reproduce on py3

>> help("modules anything")

Traceback (most recent call last):
...

This patch works (on Py3.1a1).

Amaury, are you still o.k with catching Exception rather
than just (SyntaxError, UnicodeDecodeError, ImportError)?

The code-refactoring does clean the structure up well.

Cheers,

Jervis

This appears to be the same issue as described in 5453 in which I
supplied a patch for release30-maint and py3k branches.

It's probably useful to address both bpo-4540 & 5453 at the same time.

I would have added to this issue, but I didn't see it when I raised 5453.

On a side note, the line importer.find_module(modname) in pydoc.py line
1925 appears that it may return None. If this is the case then the None
return value should be handled.

I faced similar issue. The solution is either to add tests as an
exception in pydoc or to move tests to /usr/share or /usr/share/doc.

I favor the second solution

This was fixed together with bpo-5453

Navigation Menu

Repository navigation

typo in a module describes utf-8 as uft-8 #48790

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

typo in a module describes utf-8 as uft-8 #48790

Description

Activity

johnweldon commented on Dec 4, 2008

amauryfa commented on Dec 5, 2008

malemburg commented on Dec 5, 2008

amauryfa commented on Dec 5, 2008

malemburg commented on Dec 5, 2008

amauryfa commented on Dec 5, 2008

malemburg commented on Dec 5, 2008

amauryfa commented on Dec 5, 2008

malemburg commented on Dec 5, 2008

jdwhitley commented on Mar 10, 2009

dariusp commented on Mar 10, 2009

djinn commented on May 12, 2009

amauryfa commented on Jul 15, 2009

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions