Out of source tree builds failing on main - test_importlib others unreliable

@warsaw

BPO	45506
Nosy	@warsaw, @gpshead, @ned-deily, @ericsnowcurrently, @zooba
PRs	bpo-45506: Normalize _PyPathConfig.stdlib_dir when calculated. #29040 bpo-45506: Go back to not running most of test_embed in out-of-tree builds. #29063 bpo-45506: Re-enable test_embed. #29300 bpo-45506: Fix test_embed expecting to not find stdlib in source tree build when stdlib has been installed #29649

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ericsnowcurrently'
closed_at = <Date 2021-10-22.23:23:14.310>
created_at = <Date 2021-10-18.07:53:48.061>
labels = ['type-bug', 'build', '3.11']
title = 'Out of source tree builds failing on main - test_importlib others unreliable'
updated_at = <Date 2021-11-20.00:11:43.708>
user = 'https://github.com/gpshead'

bugs.python.org fields:

activity = <Date 2021-11-20.00:11:43.708>
actor = 'steve.dower'
assignee = 'eric.snow'
closed = True
closed_date = <Date 2021-10-22.23:23:14.310>
closer = 'gregory.p.smith'
components = ['Build']
creation = <Date 2021-10-18.07:53:48.061>
creator = 'gregory.p.smith'
dependencies = []
files = []
hgrepos = []
issue_num = 45506
keywords = ['patch']
message_count = 16.0
messages = ['404159', '404160', '404161', '404167', '404188', '404191', '404192', '404222', '404226', '404345', '404838', '404839', '404842', '405632', '406629', '406632']
nosy_count = 5.0
nosy_names = ['barry', 'gregory.p.smith', 'ned.deily', 'eric.snow', 'steve.dower']
pr_nums = ['29040', '29063', '29300', '29649']
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue45506'
versions = ['Python 3.11']

The behavior is odd, I don't really know how to characterize it other than something serious has gone wrong. Memory corruption thus differing failures perhaps? it seems like maybe a race condition?

on the main branch (i cannot reproduce this on 3.10 thankfully):

git checkout https://github.com/python/cpython.git upstream
mkdir b-u && cd b-u
../upstream/configure --with-pydebug && make -j3 && ./python -m test.regrtest test_importlib

sometimes it hangs forever.
sometimes it crashes with a long list of error messages coming out of regrtest itself. The crashes appear to happen in a child process so regrtest can continue and run other tests if you tell it to run more. Sometimes it'll be an error about "import _frozenimport_lib as bootstrap" failing with sys.modules containing None. (Linux does that for me) On macOS when it doesn't hang, I get a blowup from test_importlib/test_threaded_import.py.

I've reproduced this on Linux and macOS. macOS alternates between a traceback blowup and hang with an occasional pass. On Linux it is quite consistently a huge chain of stacktraces.

Linux:

0:00:00 load avg: 2.48 Run tests sequentially
0:00:00 load avg: 2.48 [1/1] test_importlib
Failed to import test module: test.test_importlib.builtin.test_finder
Traceback (most recent call last):
  File "/home/greg/oss/python/cpython/gpshead/Lib/importlib/__init__.py", line 16, in <module>
    import _frozen_importlib as _bootstrap
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: import of _frozen_importlib halted; None in sys.modules

During handling of the above exception, another exception occurred:

...  tons more  ...

Our buildbot fleet is not bloody looking. Which is why I tried it on two different systems and OSes before reporting. When I do a ./configure and run everything within the source tree it does not fail. This is only for proper out-of-tree builds (which are what I always use - as should everybody).

Do our buildbots only do in-tree builds?

for easy pasting:

git clone https://github.com/python/cpython.git upstream && \
mkdir b-u && cd b-u && \
../upstream/configure --with-pydebug && make -j3 && \
./python -m test.regrtest test_importlib

[change python to python.exe for macos, adjust -j as appropriate]

(if I can figure out git bisect command i'll start one before going to bed...)

(oh hey, i've got macOS producing the same error as Linux now somehow)

further down in the tracebacks in that situation:

  File "/Users/greg/oss/python/upstream/Lib/importlib/_bootstrap.py", line 887, in _fix_up_module
    assert module.__file__ == __file__, (module.__file__, __file__)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: ('/Users/greg/oss/python/upstream/Lib/zipimport.py', '/Users/greg/oss/python/b-u/../upstream/Lib/zipimport.py')

git bisect on Linux yielded this culprit change:

79cf20e48d0b5d69d9fac2a0204b5ac2c366066a is the first bad commit
commit 79cf20e48d0b5d69d9fac2a0204b5ac2c366066a
Author: Eric Snow <ericsnowcurrently@gmail.com>
Date:   Thu Oct 14 15:32:18 2021 -0600

    bpo-21736: Set __file__ on frozen stdlib modules. (gh-28656)

...
Running a git bisect ritual on macOS wasn't helpful, test_importlib out of tree too often simply hangs there (going much further back). Apparently more than one possible issue at play here. Lets let this issue be for the Linux one.

I'm looking into it.

Do our buildbots only do in-tree builds?

From what I understand, we do not have any (stable?) buildbots that check out-of-tree builds.

FWIW, in the last month I started testing my changes out-of-tree before merging, after Victor once pointed out that I had broken such builds. Clearly there must be a gap in what I'm doing.

@gps

@gps, on macOS, you may need to bump the open file descriptors limit for the test process, i.e. ulimit -n 1000. I build and run out-of-tree builds all the time on macOS.

The PR I put up should fix the problem. If it doesn't then I'll probably drop the checks in the second half of FrozenImporter._fix_module().

FWIW, Greg's test case does not fail for me with 6a533a4

New changeset f4b1244 by Eric Snow in branch 'main':
bpo-45506: Go back to not running most of test_embed in out-of-tree builds. (gh-29063)
f4b1244

New changeset 17c6104 by Eric Snow in branch 'main':
bpo-45506: Normalize _PyPathConfig.stdlib_dir when calculated. (bpo-29040)
17c6104

i'm considering this fixed based on my testing. we still lack buildbots using out of tree setup. i'll ponder that. But this issue need not track that infrastructure.

Thanks for bringing this up, Greg!

New changeset 7b43828 by Eric Snow in branch 'main':
bpo-45506: Stop skipping test_embed. (gh-29300)
7b43828

Hijacking this issue number for a related test failure I found: if you "make install" Python and then run the test_embed tests from the build directory, some will successfully find the stdlib under PREFIX and fail because they expect to never find it.

PR incoming.

New changeset 4c61691 by Steve Dower in branch 'main':
bpo-45506: Fix test_embed expecting to not find stdlib in source tree build when stdlib has been installed. (GH-29649)
4c61691

Navigation Menu

Repository navigation

Out of source tree builds failing on main - test_importlib others unreliable #89669

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Out of source tree builds failing on main - test_importlib others unreliable #89669

Description

Activity

gpshead commented on Oct 18, 2021

gpshead commented on Oct 18, 2021

gpshead commented on Oct 18, 2021

gpshead commented on Oct 18, 2021

ericsnowcurrently commented on Oct 18, 2021

ericsnowcurrently commented on Oct 18, 2021

ned-deily commented on Oct 18, 2021

ericsnowcurrently commented on Oct 18, 2021

warsaw commented on Oct 18, 2021

ericsnowcurrently commented on Oct 19, 2021

ericsnowcurrently commented on Oct 22, 2021

gpshead commented on Oct 22, 2021

ericsnowcurrently commented on Oct 22, 2021

ericsnowcurrently commented on Nov 3, 2021

zooba commented on Nov 19, 2021

zooba commented on Nov 20, 2021

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions