Repository navigation
compileall.compile_dir(..., ddir="<prefix>") omits the intermediate package paths when prepending the prefix #83950
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2020-03-01.19:07:52.955>
created_at = <Date 2020-02-27.05:18:05.962>
labels = ['3.7', '3.8', 'type-bug', 'library', '3.9']
title = 'compileall.compile_dir(..., ddir="<prefix>") omits the intermediate package paths when prepending the prefix'
updated_at = <Date 2020-03-01.19:07:52.954>
user = 'https://github.com/gpshead'bugs.python.org fields:
activity = <Date 2020-03-01.19:07:52.954>
actor = 'gregory.p.smith'
assignee = 'none'
closed = True
closed_date = <Date 2020-03-01.19:07:52.955>
closer = 'gregory.p.smith'
components = ['Library (Lib)']
creation = <Date 2020-02-27.05:18:05.962>
creator = 'gregory.p.smith'
dependencies = []
files = []
hgrepos = []
issue_num = 39769
keywords = ['patch']
message_count = 11.0
messages = ['362767', '362768', '362782', '362785', '362787', '362788', '362956', '362961', '363077', '363078', '363079']
nosy_count = 1.0
nosy_names = ['gregory.p.smith']
pr_nums = ['18676', '18718', '18725']
priority = 'normal'
resolution = 'fixed'
stage = 'commit review'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue39769'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9']
Activity
gpshead commentedon Feb 27, 2020
Easiest to demonstrate as such:
Run that script and you'll see:
Listing 'bug'...
Compiling 'bug/init.py'...
Listing 'bug/foo'...
Compiling 'bug/foo/init.py'...
Compiling 'bug/foo/bar.py'...
Listing bug ...
Compiling bug/init.py ...
Listing bug/pycache ...
Listing bug/foo ...
Compiling bug/foo/init.py ...
Listing bug/foo/pycache ...
Compiling bug/foo/bar.py ...
prefix embedded in PY3 pyc code object for lib.foo.bar:
<prefix>/bar.py
prefix embedded in PY2 pyc code object for lib.foo.bar:
<prefix>/foo/bar.pyt
Notice that the Python 3 pyc file contains a code.co_filename of "<prefix>/bar.py" instead of the correct value (that Python 2 inserts) of "<prefix>/foo/bar.py".
gpshead commentedon Feb 27, 2020
This has probably been happening for many 3 releases. I noticed it on 3.6 when investigating a problem with wrong paths in tracebacks coming from code which we used compileall.compile_dir(src_root, quiet=1, ddir="<prefix">) on.
I'm guessing ddir= (-d on the cli) isn't widely used?
gpshead commentedon Feb 27, 2020
Ironically 3.9a4 gains some compileall enhancements that could be used as an awkward workaround from https://bugs.python.org/issue38112:
python 2.7 compile_dir(d, ddir="<foo>") behavior can be had with
python 3.9a4 compile_dir(d, prependdir="<foo>", stripdir=d)
I still intend to fix the actual ddir behavior as it is wrong.
gpshead commentedon Feb 27, 2020
The regression was caused by the parallelization enhancement implementation from https://bugs.python.org/issue16104
gpshead commentedon Feb 27, 2020
meaning this regression was introduced in 3.5.
gpshead commentedon Feb 27, 2020
belated postmortem: if there had been tests against the filename encoded in the pycs living in subdirs as generated by compile_dir, this regression would not have happened.
gpshead commentedon Feb 29, 2020
New changeset 0267335 by Gregory P. Smith in branch 'master':
bpo-39769: Fix compileall ddir for subpkgs. (GH-18676)
0267335
gpshead commentedon Feb 29, 2020
keeping this open while i investigate if fixing 3.8 and 3.7 is feasible. if nothing else i'll try to add a note to the docs about the issue in 3.5-3.8 with a code sample suggested workaround.
gpshead commentedon Mar 1, 2020
New changeset ce720d3 by Gregory P. Smith in branch '3.8':
bpo-39769: Fix compileall ddir for subpkgs. (GH-18676) (GH-18718)
ce720d3
gpshead commentedon Mar 1, 2020
New changeset 7c64726 by Gregory P. Smith in branch '3.7':
[3.7] bpo-39769: Fix compileall ddir for subpkgs. (GH-18676) (GH-18718) (GH-18725)
7c64726
gpshead commentedon Mar 1, 2020
if anyone needs this on their older 3.6 or 3.5 trees, the 3.7/3.8 patch is a trivial backport.