Skip to content

Out of tree build of Python 3.11.0a2+ breaks regen-frozen #90024

Closed
Closed
Out of tree build of Python 3.11.0a2+ breaks regen-frozen#90024
Labels
3.11only security fixesbuildThe build process and cross-build

Description

@hroncok
hroncok
mannequin
opened on Nov 22, 2021
BPO 45866
Nosy @vstinner, @encukou, @ericsnowcurrently, @hroncok, @miss-islington
PRs
  • bpo-45866: pegen strips directory of "generated from" header #29777
  • [3.10] bpo-45866: pegen strips directory of "generated from" header (GH-29777) #29792
  • [3.9] bpo-45866: pegen strips directory of "generated from" header (GH-29777) (GH-29792) #29797
  • bpo-45866: Fix typo in the NEWS entry #29798
  • [3.10] bpo-45866: Fix typo in the NEWS entry (GH-29798) #29803
  • [3.9] bpo-45866: Fix typo in the NEWS entry (GH-29798) #29804
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2021-11-26.16:30:15.411>
created_at = <Date 2021-11-22.11:32:02.800>
labels = ['build', '3.11']
title = 'Out of tree build of Python 3.11.0a2+ breaks regen-frozen'
updated_at = <Date 2021-11-26.22:20:24.860>
user = 'https://github.com/hroncok'

    bugs.python.org fields:

    activity = <Date 2021-11-26.22:20:24.860>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2021-11-26.16:30:15.411>
closer = 'vstinner'
components = ['Build']
creation = <Date 2021-11-22.11:32:02.800>
creator = 'hroncok'
dependencies = []
files = []
hgrepos = []
issue_num = 45866
keywords = ['patch']
message_count = 10.0
messages = ['406764', '406766', '406769', '407039', '407057', '407064', '407067', '407099', '407101', '407102']
nosy_count = 5.0
nosy_names = ['vstinner', 'petr.viktorin', 'eric.snow', 'hroncok', 'miss-islington']
pr_nums = ['29777', '29792', '29797', '29798', '29803', '29804']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue45866'
versions = ['Python 3.11']

    Activity

    1. hroncok commented on Nov 22, 2021

      @hroncok
      hroncokmannequin
      on Nov 22, 2021
      MannequinAuthor

      In Fedora, when building Python 3.11.0a2 with Python 3.11.0a2 for regen, we found that regen-frozen fails.

      It can be reproduced either on the v3.11.0a2 tag or on the main branch.

      On tag v3.11.0a2 with 3.11.0a2 installed in $PATH:

      # in root of cpython local clone:
      $ git clean -fdx
      $ mkdir -p build/optimized
      $ cd build/optimized
      $ ../../configure
      $ make regen-all PYTHON_FOR_REGEN=python3.11
      ...
      ERROR: missing _freeze_module

      On the main branch:

      # in root of cpython local clone:
      $ git clean -fdx
      $ mkdir -p build/optimized
      $ cd build/optimized
      $ ../../configure
      $ make
      $ make regen-all PYTHON_FOR_REGEN=./python
      (success)

      However, the working tree is dirty:

      $ git diff
diff --git a/Parser/parser.c b/Parser/parser.c
index b3aa35989ed..c26cd6eeb05 100644
--- a/Parser/parser.c
+++ b/Parser/parser.c
@@ -1,4 +1,4 @@
-// @generated by pegen from ./Grammar/python.gram
+// @generated by pegen from ../../Grammar/python.gram
 #include "pegen.h"
 
 #if defined(Py_DEBUG) && defined(Py_BUILD_CORE)
diff --git a/Tools/peg_generator/pegen/grammar_parser.py b/Tools/peg_generator/pegen/grammar_parser.py
index 6e9f7d3d11d..fbbbfad76b0 100644
--- a/Tools/peg_generator/pegen/grammar_parser.py
+++ b/Tools/peg_generator/pegen/grammar_parser.py
@@ -1,5 +1,5 @@
 #!/usr/bin/env python3.8
-# @generated by pegen from ./Tools/peg_generator/pegen/metagrammar.gram
+# @generated by pegen from ../../Tools/peg_generator/pegen/metagrammar.gram
 
 import ast
 import sys

      And if we install Python somewhere:

      $ make
$ make install DESTDIR=/tmp/python

      And use that one again for regen:

      $ cd ../..
$ git clean -fdx
$ mkdir -p build/optimized
$ cd build/optimized
$ ../../configure
$ make regen-all PYTHON_FOR_REGEN=/tmp/python/usr/local/bin/python3.11
...
# Regenerate Lib/keyword.py from Grammar/python.gram and Grammar/Tokens
# using Tools/peg_generator/pegen
PYTHONPATH=../../Tools/peg_generator /tmp/python/usr/local/bin/python3.11 -m pegen.keywordgen \
	../../Grammar/python.gram \
	../../Grammar/Tokens \
	../../Lib/keyword.py.new
/tmp/python/usr/local/bin/python3.11 ../../Tools/scripts/update_file.py ../../Lib/keyword.py ../../Lib/keyword.py.new
/tmp/python/usr/local/bin/python3.11 ../../Tools/scripts/freeze_modules.py
ERROR: missing _freeze_module
make: *** [Makefile:1259: regen-frozen] Error 1

      It fails. I've isolated the failure to:

      $ make regen-frozen PYTHON_FOR_REGEN=/tmp/python/usr/local/bin/python3.11
/tmp/python/usr/local/bin/python3.11 ../../Tools/scripts/freeze_modules.py
ERROR: missing _freeze_module
make: *** [Makefile:1259: regen-frozen] Error 1
    2. hroncok
      added
      3.11only security fixes
      buildThe build process and cross-build
      on Nov 22, 2021

    3. hroncok commented on Nov 22, 2021

      @hroncok
      hroncokmannequin
      on Nov 22, 2021
      MannequinAuthor

      git bisect says:

      ff8859d is the first new commit
      commit ff8859d
      Author: Victor Stinner <vstinner@python.org>
      Date: Thu Oct 7 21:19:13 2021 +0200

      bpo-45402: Fix test_tools.test_sundry() (GH-28786)

Fix test_tools.test_sundry() when Python is built out of tree: fix
how the freeze_modules.py tool locates the _freeze_module program.

      Misc/NEWS.d/next/Tests/2021-10-07-13-43-01.bpo-45402.jlQvep.rst | 3 +++
      Tools/scripts/freeze_modules.py | 7 ++++---
      2 files changed, 7 insertions(+), 3 deletions(-)
      create mode 100644 Misc/NEWS.d/next/Tests/2021-10-07-13-43-01.bpo-45402.jlQvep.rst

      It seems the fix for an out-of-tree build bug introduced another one.

    4. hroncok commented on Nov 22, 2021

      @hroncok
      hroncokmannequin
      on Nov 22, 2021
      MannequinAuthor

      It also seems that it is *not* necessary to use Python 3.11 for regen-frozen to get the failure:

      $ make regen-frozen PYTHON_FOR_REGEN=python3.10
python3.10 ../../Tools/scripts/freeze_modules.py
ERROR: missing _freeze_module
make: *** [Makefile:1259: regen-frozen] Error 1

$ make regen-frozen PYTHON_FOR_REGEN=python3.9
python3.9 ../../Tools/scripts/freeze_modules.py
ERROR: missing _freeze_module
make: *** [Makefile:1259: regen-frozen] Error 1

      My guess is that the fix for test_tools.test_sundry() actually broke the real usage. It has:

              # When building out of the source tree, get the tool from directory
        # of the Python executable
        TOOL = os.path.dirname(sys.executable)
        TOOL = os.path.join(TOOL, 'Programs', '_freeze_module')

      But sys.executable is *not* the Python we have built, but the Python we use for regen.
      E.g. when Python for regen is /usr/bin/python3.X, it tries to use /usr/bin/Programs/_freeze_module which obviously is not there.

      The assumption that sys.executable is the freshly built Python is only correct in the tests, but not in reality.

    5. vstinner commented on Nov 26, 2021

      @vstinner
      vstinner
      on Nov 26, 2021
      Member

      New changeset 253b7a0 by Victor Stinner in branch 'main':
      bpo-45866: pegen strips directory of "generated from" header (GH-29777)
      253b7a0

    6. vstinner commented on Nov 26, 2021

      @vstinner
      vstinner
      on Nov 26, 2021
      Member

      New changeset b6defde by Victor Stinner in branch '3.10':
      bpo-45866: pegen strips directory of "generated from" header (GH-29777) (GH-29792)
      b6defde

    7. vstinner commented on Nov 26, 2021

      @vstinner
      vstinner
      on Nov 26, 2021
      Member

      New changeset 93a540d by Victor Stinner in branch '3.9':
      bpo-45866: pegen strips directory of "generated from" header (GH-29777) (GH-29792) (GH-29797)
      93a540d

    8. vstinner commented on Nov 26, 2021

      @vstinner
      vstinner
      on Nov 26, 2021
      Member

      It seems like all bugs reported in this issue has been fixed, so I close the issue. Thanks for the bug report ;-)

      -# @generated by pegen from ./Tools/peg_generator/pegen/metagrammar.gram
      +# @generated by pegen from ../../Tools/peg_generator/pegen/metagrammar.gram

      This issue is now fixed in 3.9, 3.10 and main branches.

      ERROR: missing _freeze_module

      I'm no longer able to reproduce this error on an up-to-date main branch. The issue has been fixed by this change:

      commit 5be98e5
      Author: Guido van Rossum <guido@python.org>
      Date: Tue Nov 23 08:56:06 2021 -0800

      bpo-45873: Get rid of bootstrap_python (bpo-29717)

Instead we use $(PYTHON_FOR_REGEN) .../deepfreeze.py with the
frozen .h file as input, as we did for Windows in bpo-45850.

We also get rid of the code that generates the .h files
when make regen-frozen is run (i.e., .../make_frozen.py),
and the MANIFEST file.

Restore Python 3.8 and 3.9 as Windows host Python again

Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
    9. vstinner
      closed this as completedon Nov 26, 2021
    10. vstinner
      closed this as completedon Nov 26, 2021

    11. vstinner commented on Nov 26, 2021

      @vstinner
      vstinner
      on Nov 26, 2021
      Member

      New changeset 9a7611a by Victor Stinner in branch 'main':
      bpo-45866: Fix typo in the NEWS entry (GH-29798)
      9a7611a

    12. miss-islington commented on Nov 26, 2021

      @miss-islington
      miss-islington
      on Nov 26, 2021
      Contributor

      New changeset fc16ea9 by Miss Islington (bot) in branch '3.10':
      bpo-45866: Fix typo in the NEWS entry (GH-29798)
      fc16ea9

    13. miss-islington commented on Nov 26, 2021

      @miss-islington
      miss-islington
      on Nov 26, 2021
      Contributor

      New changeset b3f14da by Miss Islington (bot) in branch '3.9':
      bpo-45866: Fix typo in the NEWS entry (GH-29798)
      b3f14da

    14. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.11only security fixesbuildThe build process and cross-build

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions