3.0.1 crashes in unicode path

@pitrou

BPO	5273
Nosy	@pitrou
Files	fix_import.patch: for py3k branch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2009-03-04.01:58:45.260>
created_at = <Date 2009-02-15.11:26:12.722>
labels = ['interpreter-core', 'expert-unicode', 'OS-windows', 'type-crash']
title = '3.0.1 crashes in unicode path'
updated_at = <Date 2010-04-27.20:32:23.571>
user = 'https://bugs.python.org/miwa'

bugs.python.org fields:

activity = <Date 2010-04-27.20:32:23.571>
actor = 'loewis'
assignee = 'none'
closed = True
closed_date = <Date 2009-03-04.01:58:45.260>
closer = 'ocean-city'
components = ['Interpreter Core', 'Unicode', 'Windows']
creation = <Date 2009-02-15.11:26:12.722>
creator = 'miwa'
dependencies = []
files = ['13101']
hgrepos = []
issue_num = 5273
keywords = ['patch']
message_count = 9.0
messages = ['82150', '82152', '82153', '82154', '82159', '82160', '82164', '83110', '83113']
nosy_count = 3.0
nosy_names = ['pitrou', 'ocean-city', 'miwa']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue5273'
versions = ['Python 3.0', 'Python 3.1']

In unicode path Python 3.0.1 crashes when importing compiled module.
This does not happen on Python 3.0, new in 3.0.1.

Detailed Situation:
OS: win2000
current pathname contains Japanese characters.
./a.py contains only a statement "import b".
./b.py is empty.
> python a.py
(nothing is happen but b.pyc is created)
> python a.py
Traceback (most recent call last):
  File "a.py", line 1, in <module>
    import b
UnicodeDecodeError: 'utf8' codec can't decode byte 0x82 in position 3:
unexpected code byte

Quick observation. This bug was introduces in r68363.

import.c(994)
	newname = PyUnicode_FromString(pathname);

pathname is mbcs on windows, but PyUnicode_FromString assumes it as UTF8.

Here is a patch.

Gasp. Sorry for the bug.
Should PyUnicode_CompareWithASCIIString() be replaced with something
else as well?

I'm not sure. Even my patch might not be correct anyway.

On my VC6 Debugger,
update_compiled_module(PyCodeObject *co, char *pathname)
pathname is surely mbcs.

But its caller load_source_module is calling

	if (fstat(fileno(fp), &st) != 0) {
		PyErr_Format(PyExc_RuntimeError,
			     "unable to get file status from '%s'",
			     pathname);
		return NULL;
	}

I've looked into PyErr_Format code, it seems %s assumes utf-8. Anway,
it's difficult to know char* is utf-8 or filesystem encoding. :-(

I tracked down, and I found this mbcs path is set in Python/import.c(1394)
find_module.

	if (PyUnicode_Check(v)) {
		v = PyUnicode_AsEncodedString(v, 
		    Py_FileSystemDefaultEncoding, NULL);
		if (v == NULL)
			return NULL;
	}

And this was introduced in r64126 to fix segfault mentioned in
bpo-1342. I'm not understanding why segfault happened but, I feel this
issue is the part of big problem. (bpo-3080)

Should PyUnicode_CompareWithASCIIString() be replaced with something
else as well?

I hope revised patch will fix this too. There seems to be no function to
compare unicode object and file system encoded string, so I moved
unicode creation before comparation. This might increase overhead a bit.

bpo-3080 is big issue, so this is minimal solution for this issue. I
confirmed test_import.py passed.

I cannot say anything except that the patch looks ok. If it doesn't make
anything worse and solves the present problem, I guess you can commit it.

Thanks, fixed in r70157(py3k) and r70158(release30-maint)

Navigation Menu

Repository navigation

3.0.1 crashes in unicode path #49523

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

3.0.1 crashes in unicode path #49523

Description

Activity

miwa commented on Feb 15, 2009

ocean-city commented on Feb 15, 2009

ocean-city commented on Feb 15, 2009

pitrou commented on Feb 15, 2009

ocean-city commented on Feb 15, 2009

ocean-city commented on Feb 15, 2009

ocean-city commented on Feb 15, 2009

pitrou commented on Mar 3, 2009

ocean-city commented on Mar 4, 2009

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions