3.10's make regen-pegen-metaparser fails with 3.12+

On the 3.10 branch:

$ PYTHON_FOR_REGEN=/usr/bin/python3.11 make regen-pegen-metaparser 
PYTHONPATH=./Tools/peg_generator /usr/bin/python3.11 -m pegen -q python \
./Tools/peg_generator/pegen/metagrammar.gram \
-o ./Tools/peg_generator/pegen/grammar_parser.py.new
/usr/bin/python3.11 ./Tools/scripts/update_file.py ./Tools/peg_generator/pegen/grammar_parser.py \
./Tools/peg_generator/pegen/grammar_parser.py.new

$ PYTHON_FOR_REGEN=/usr/bin/python3.12 make regen-pegen-metaparser 
PYTHONPATH=./Tools/peg_generator /usr/bin/python3.12 -m pegen -q python \
./Tools/peg_generator/pegen/metagrammar.gram \
-o ./Tools/peg_generator/pegen/grammar_parser.py.new
  File "./Tools/peg_generator/pegen/metagrammar.gram", line 87
    | NAME '[' type=NAME '*' ']' '=' ~ item {NamedItem(name.string, item, f"{type.string}*")}
                                                                          ^
SyntaxError: pegen parse failure
For full traceback, use -v
make: *** [Makefile:854: regen-pegen-metaparser] Error 1

This now makes PR CI fail, see e.g. https://github.com/python/cpython/actions/runs/11274132145/job/31352641619?pr=125255

Linked PRs

[3.10] gh-125529: Avoid f-strings in the metagrammar #125582

That's because of PEP 701 and the fact that f-strings are no longer STRING tokens (grammar actions need to be valid Python tokens and the accepted tokens need to be listed in the actions mini-grammar).

This patch solves the issue, though I don't know how we would fix this now that 3.10 is security-only:

diff --git a/Tools/peg_generator/pegen/grammar_parser.py b/Tools/peg_generator/pegen/grammar_parser.py
index 892df5cf8cc..de735649d46 100644
--- a/Tools/peg_generator/pegen/grammar_parser.py
+++ b/Tools/peg_generator/pegen/grammar_parser.py
@@ -420,7 +420,7 @@ def named_item(self) -> Optional[NamedItem]:
+++ b/Tools/peg_generator/pegen/grammar_parser.py
@@ -420,7 +420,7 @@ def named_item(self) -> Optional[NamedItem]:
             and
             (item := self.item())
         ):
-            return NamedItem ( name . string , item , f"{type.string}*" )
+            return NamedItem ( name . string , item , type . string + "*" )
         self.reset(mark)
         if cut: return None
         cut = False
diff --git a/Tools/peg_generator/pegen/metagrammar.gram b/Tools/peg_generator/pegen/metagrammar.gram
index bb4355fd189..77695d5e0a9 100644
--- a/Tools/peg_generator/pegen/metagrammar.gram
+++ b/Tools/peg_generator/pegen/metagrammar.gram
@@ -84,7 +84,7 @@ items[NamedItemList]:
     | named_item { [named_item] }
 
 named_item[NamedItem]:
-    | NAME '[' type=NAME '*' ']' '=' ~ item {NamedItem(name.string, item, f"{type.string}*")}
+    | NAME '[' type=NAME '*' ']' '=' ~ item {NamedItem(name.string, item, type.string+"*")}
     | NAME '[' type=NAME ']' '=' ~ item {NamedItem(name.string, item, type.string)}
     | NAME '=' ~ item {NamedItem(name.string, item)}
     | item {NamedItem(None, item)}

Thank you!

Ultimately that's the release manager's call, but I think this is the right solution. I'll prepare a PR and write some reasoning there.

[3.10] pythongh-125529: Avoid f-strings in the metagrammar

[3.10] gh-125529: Avoid f-strings in the metagrammar (#125582)

Navigation Menu

Repository navigation

3.10's `make regen-pegen-metaparser` fails with 3.12+ #125529

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

3.10's make regen-pegen-metaparser fails with 3.12+ #125529

Description

Linked PRs

Activity

lysnikolaou commented on Oct 15, 2024

encukou commented on Oct 16, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

3.10's `make regen-pegen-metaparser` fails with 3.12+ #125529