Repository navigation
PR check "Check if generated files are up to date" failing intermittently #91302
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/ericsnowcurrently'
closed_at = <Date 2022-04-01.15:37:10.755>
created_at = <Date 2022-03-28.16:50:08.590>
labels = ['build', '3.11']
title = 'PR check "Check if generated files are up to date" failing intermittently'
updated_at = <Date 2022-04-01.15:37:10.755>
user = 'https://github.com/ericsnowcurrently'bugs.python.org fields:
activity = <Date 2022-04-01.15:37:10.755>
actor = 'eric.snow'
assignee = 'eric.snow'
closed = True
closed_date = <Date 2022-04-01.15:37:10.755>
closer = 'eric.snow'
components = ['Build']
creation = <Date 2022-03-28.16:50:08.590>
creator = 'eric.snow'
dependencies = []
files = []
hgrepos = []
issue_num = 47146
keywords = ['patch']
message_count = 13.0
messages = ['416193', '416201', '416211', '416214', '416379', '416396', '416410', '416443', '416444', '416454', '416456', '416457', '416496']
nosy_count = 4.0
nosy_names = ['christian.heimes', 'Mark.Shannon', 'eric.snow', 'brandtbucher']
pr_nums = ['32162', '32206', '32218']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue47146'
versions = ['Python 3.11']
Activity
ericsnowcurrently commentedon Mar 28, 2022
The "Check if generated files are up to date" GitHub check for PRs has been failing recently. It may also impact local usage of "make regen-all".
Example: https://github.com/python/cpython/runs/5719012664
This may be related to #76242.
ericsnowcurrently commentedon Mar 28, 2022
There's probably something racy with make.
See: https://github.com/python/cpython/runs/5712538599?check_suite_focus=true#step:10:1147
ericsnowcurrently commentedon Mar 28, 2022
New changeset 4c116f7 by Eric Snow in branch 'main':
bpo-47146: Eliminate a race between make regen-deepfreeze and make regen-global-objects. (gh-32162)
4c116f7
ericsnowcurrently commentedon Mar 28, 2022
Looks like that fixed it, per #32134.
ericsnowcurrently commentedon Mar 30, 2022
Looks like this is still an intermittent problem:
make regen-limited-abias generated #32195+ failed: https://github.com/python/cpython/runs/5756616733
+ failed: https://github.com/python/cpython/runs/5753267869
+ failed: https://github.com/python/cpython/runs/5757169625
+ failed: https://github.com/python/cpython/runs/5756616733
+ passed: https://github.com/python/cpython/runs/5757213346
+ failed: ...
+ passed: https://github.com/python/cpython/runs/5757178754
Per Mark Shannon (on discord):
The "Check if generated files are up to date" is still failing consistently. It looks like the makefile is missing a dependency on the ./Programs/_freeze_module for targets that require /Programs/_freeze_module
tiran commentedon Mar 30, 2022
The problem is likely a recursive make call "$(MAKE)" in combination with parallel execution. As far as I understand it, GNU make cannot track dependencies properly, which can lead to partial or invalid dependency graphs. One make process can modify a file while another uses the file. The permission error or ETXTBSY error are manifestation of one process creating an executable while another process is attempting to execve() it.
The paper https://accu.org/journals/overload/14/71/miller_2004/ explains the issues with recursion in great detail.
ericsnowcurrently commentedon Mar 31, 2022
New changeset db4dada by Eric Snow in branch 'main':
bpo-47146: Avoid Using make Recursively (gh-32206)
db4dada
ericsnowcurrently commentedon Mar 31, 2022
I re-ran jobs that had failed before I merged that #76387. Several passed, but the following are still failing:
+ https://github.com/python/cpython/runs/5773938424
+ https://github.com/python/cpython/runs/5774054192
+ https://github.com/python/cpython/runs/5773949869
Back to the drawing board...
ericsnowcurrently commentedon Mar 31, 2022
Brandt pointed out this is consistently reproducible locally:
make clean regen-all -j
I'll get this sorted out today.
ericsnowcurrently commentedon Mar 31, 2022
New changeset e7bb7c2 by Eric Snow in branch 'main':
bpo-47146: Stop Depending On regen-deepfreeze For regen-global-objects (gh-32218)
e7bb7c2
ericsnowcurrently commentedon Mar 31, 2022
I'll keep an eye on PRs for the next day or so.
ericsnowcurrently commentedon Mar 31, 2022
Specifically: https://github.com/python/cpython/actions/workflows/build.yml.
ericsnowcurrently commentedon Apr 1, 2022
Looks like #76399 worked.