Skip to content

PR check "Check if generated files are up to date" failing intermittently #91302

Closed
Closed
PR check "Check if generated files are up to date" failing intermittently#91302
Assignees
ericsnowcurrently
Labels
3.11only security fixesbuildThe build process and cross-build

Description

@ericsnowcurrently
ericsnowcurrently
opened on Mar 28, 2022
BPO 47146
Nosy @tiran, @markshannon, @ericsnowcurrently, @brandtbucher
PRs
  • bpo-47146: Eliminate a Race Between make regen-deepfreeze and make regen-global-objects #32162
  • bpo-47146: Avoid Using make Recursively #32206
  • bpo-47146: Stop Depending On regen-deepfreeze For regen-global-objects #32218
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/ericsnowcurrently'
closed_at = <Date 2022-04-01.15:37:10.755>
created_at = <Date 2022-03-28.16:50:08.590>
labels = ['build', '3.11']
title = 'PR check "Check if generated files are up to date" failing intermittently'
updated_at = <Date 2022-04-01.15:37:10.755>
user = 'https://github.com/ericsnowcurrently'

    bugs.python.org fields:

    activity = <Date 2022-04-01.15:37:10.755>
actor = 'eric.snow'
assignee = 'eric.snow'
closed = True
closed_date = <Date 2022-04-01.15:37:10.755>
closer = 'eric.snow'
components = ['Build']
creation = <Date 2022-03-28.16:50:08.590>
creator = 'eric.snow'
dependencies = []
files = []
hgrepos = []
issue_num = 47146
keywords = ['patch']
message_count = 13.0
messages = ['416193', '416201', '416211', '416214', '416379', '416396', '416410', '416443', '416444', '416454', '416456', '416457', '416496']
nosy_count = 4.0
nosy_names = ['christian.heimes', 'Mark.Shannon', 'eric.snow', 'brandtbucher']
pr_nums = ['32162', '32206', '32218']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue47146'
versions = ['Python 3.11']

    Activity

    1. ericsnowcurrently commented on Mar 28, 2022

      @ericsnowcurrently
      ericsnowcurrently
      on Mar 28, 2022
      MemberAuthor

      The "Check if generated files are up to date" GitHub check for PRs has been failing recently. It may also impact local usage of "make regen-all".

      Example: https://github.com/python/cpython/runs/5719012664

      This may be related to #76242.

    2. ericsnowcurrently
      added
      3.11only security fixes
      on Mar 28, 2022
    3. ericsnowcurrently
      self-assigned this
      on Mar 28, 2022
    4. ericsnowcurrently
      added
      buildThe build process and cross-build
      3.11only security fixes
      on Mar 28, 2022
    5. ericsnowcurrently
      self-assigned this
      on Mar 28, 2022
    6. ericsnowcurrently
      added
      buildThe build process and cross-build
      on Mar 28, 2022

    7. ericsnowcurrently commented on Mar 28, 2022

      @ericsnowcurrently
      ericsnowcurrently
      on Mar 28, 2022
      MemberAuthor

      There's probably something racy with make.

      See: https://github.com/python/cpython/runs/5712538599?check_suite_focus=true#step:10:1147

    8. ericsnowcurrently commented on Mar 28, 2022

      @ericsnowcurrently
      ericsnowcurrently
      on Mar 28, 2022
      MemberAuthor

      New changeset 4c116f7 by Eric Snow in branch 'main':
      bpo-47146: Eliminate a race between make regen-deepfreeze and make regen-global-objects. (gh-32162)
      4c116f7

    9. ericsnowcurrently commented on Mar 28, 2022

      @ericsnowcurrently
      ericsnowcurrently
      on Mar 28, 2022
      MemberAuthor

      Looks like that fixed it, per #32134.

    10. ericsnowcurrently
      closed this as completedon Mar 28, 2022
    11. ericsnowcurrently
      closed this as completedon Mar 28, 2022

    12. ericsnowcurrently commented on Mar 30, 2022

      @ericsnowcurrently
      ericsnowcurrently
      on Mar 30, 2022
      MemberAuthor

      Looks like this is still an intermittent problem:

      Per Mark Shannon (on discord):

      The "Check if generated files are up to date" is still failing consistently. It looks like the makefile is missing a dependency on the ./Programs/_freeze_module for targets that require /Programs/_freeze_module

    13. ericsnowcurrently
      reopened this on Mar 30, 2022
    14. ericsnowcurrently
      reopened this on Mar 30, 2022

    15. tiran commented on Mar 30, 2022

      @tiran
      tiran
      on Mar 30, 2022
      Member

      The problem is likely a recursive make call "$(MAKE)" in combination with parallel execution. As far as I understand it, GNU make cannot track dependencies properly, which can lead to partial or invalid dependency graphs. One make process can modify a file while another uses the file. The permission error or ETXTBSY error are manifestation of one process creating an executable while another process is attempting to execve() it.

      The paper https://accu.org/journals/overload/14/71/miller_2004/ explains the issues with recursion in great detail.

    16. ericsnowcurrently commented on Mar 31, 2022

      @ericsnowcurrently
      ericsnowcurrently
      on Mar 31, 2022
      MemberAuthor

      New changeset db4dada by Eric Snow in branch 'main':
      bpo-47146: Avoid Using make Recursively (gh-32206)
      db4dada

    17. ericsnowcurrently commented on Mar 31, 2022

      @ericsnowcurrently
      ericsnowcurrently
      on Mar 31, 2022
      MemberAuthor

      I re-ran jobs that had failed before I merged that #76387. Several passed, but the following are still failing:

      Back to the drawing board...

    18. ericsnowcurrently commented on Mar 31, 2022

      @ericsnowcurrently
      ericsnowcurrently
      on Mar 31, 2022
      MemberAuthor

      Brandt pointed out this is consistently reproducible locally:

      make clean regen-all -j

      I'll get this sorted out today.

    19. ericsnowcurrently commented on Mar 31, 2022

      @ericsnowcurrently
      ericsnowcurrently
      on Mar 31, 2022
      MemberAuthor

      New changeset e7bb7c2 by Eric Snow in branch 'main':
      bpo-47146: Stop Depending On regen-deepfreeze For regen-global-objects (gh-32218)
      e7bb7c2

    20. ericsnowcurrently commented on Mar 31, 2022

      @ericsnowcurrently
      ericsnowcurrently
      on Mar 31, 2022
      MemberAuthor

      I'll keep an eye on PRs for the next day or so.

    21. ericsnowcurrently commented on Mar 31, 2022

      @ericsnowcurrently
      ericsnowcurrently
      on Mar 31, 2022
      MemberAuthor

      Specifically: https://github.com/python/cpython/actions/workflows/build.yml.

    22. ericsnowcurrently commented on Apr 1, 2022

      @ericsnowcurrently
      ericsnowcurrently
      on Apr 1, 2022
      MemberAuthor

      Looks like #76399 worked.

    23. ericsnowcurrently
      closed this as completedon Apr 1, 2022
    24. ericsnowcurrently
      closed this as completedon Apr 1, 2022
    25. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    3.11only security fixesbuildThe build process and cross-build

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions