Bytecode compilation output depends on order of files compiled

This is minimal reproduction of this downstream bug report: astral-sh/uv#10619

The output of compileall.compile_file depends on the order in which the files are compiled. This means compilation is non-deterministic if builds are distributed over a process pool.

This becomes a problem when building docker images, where you usually bytecode compile ahead of time for faster startup, and where the hash of the image depends on all files in the image, including the .pyc files.

Specifically, the output of

a = {"foo", 2, 3}

def f():
    b = {"foo", 2, 3}

is different if we previously compiled another file with

import foo

Reproducer script:

#!/bin/bash

set -e

script=$(cat << EOF
import compileall
import sys

for path in sys.argv[1:]:
    compileall.compile_file(path)
EOF
)

cat << EOF > a.py
import foo
EOF

cat << EOF > b.py
a = {"foo", 2, 3}

def f():
    b = {"foo", 2, 3}
EOF

# Both files
rm -rf __pycache__
python3.14 -c "$script" a.py b.py
sha256sum __pycache__/b.cpython-314.pyc

# For debugging
cp __pycache__/b.cpython-314.pyc b1.cpython-314.pyc

# Single file only
rm -rf __pycache__
python3.14 -c "$script" b.py
sha256sum __pycache__/b.cpython-314.pyc

# For debugging
cp __pycache__/b.cpython-314.pyc b2.cpython-314.pyc

This is caused be different refcounts in the marshalled files:

import marshal
import sys

with open("b1.cpython-313.pyc", "rb") as f:
  f.read(16)  # Skip header
  pyc1 = marshal.load(f)

with open("b2.cpython-313.pyc", "rb") as f:
  f.read(16)  # Skip header
  pyc2 = marshal.load(f)

print(sys.getrefcount(pyc1.co_consts[0]))
print(sys.getrefcount(pyc2.co_consts[0]))

This prints 2 and 3.

The original report is from 3.13, i've reproduced it with 3.14.0a4. It happens at least on linux and windows.

CPython versions tested on:

3.14

Operating systems tested on:

Linux

Linked PRs

Yes, bytecode compilation is non-deterministic. See also this issue: #78274 (comment)

A PR to make it deterministic without sacrificing performance would be welcome. You can also use an external tool to post-process the pyc.

Is it valid to modify the serialized refcounts (to some specific value)? I'm having trouble following what the marshalled refcounts mean and when they'd be DECREF'd.

They're not refcounts. FLAG_REF gets added if an object might appear later in the file, in which case it can later be referenced by TYPE_REF (using an index to the array of all objects with FLAG_REF).

One solution would be to flag all objects (inflating the array and the indices), or doing a second pass over the pyc (wasteful if not needed, so CPython doesn't do it)... or something like redesigning the serialization format entirely.

pythongh-129724: Stabilize marshal refleak regression

pythongh-129724: Limit constant cache changes to free-threaded builds

One specific case of this I stumbled across while analyzing an image build given the way bazel's rules_python works by keeping a long running python process to feed things to compile to:

What: Take a module in which two functions contain the same constant set-membership test, e.g. if kind in {"alpha_kind", "beta_kind"}:. Compile it twice with compile() + marshal.dumps() (which is what py_compile, compileall and importlib do). If, during one of those compiles, some other live code object in the process uses alpha_kind or beta_kind as a name or string constant, the resulting bytes differ from the other compile by a handful of bytes. The two code objects compare equal and behave identically; only the serialization differs.

Why? The compiler deduplicates equal constants within a compilation unit, so both functions share one frozenset object. When each function's code object is created, intern_constants() interns identifier-like strings inside that frozenset; because frozensets are immutable, if any element gets replaced by an already-interned string it builds a brand-new frozenset for that one code object, silently undoing the sharing. Whether an element is "already interned" depends on the rest of the process: string constants of other live code objects (mortal, so this tracks liveness and thread timing) and, on 3.12+, every name any code object in the process has ever had (co_names are interned immortally, so this tracks the process's whole import and compile history). marshal then exposes object identity: w_ref() emits a back-reference for an object seen before only if its refcount is > 1, so "one shared frozenset" and "two equal frozensets" serialize differently.

Possible suggestion for that one: Intern eligible string constants when the compiler first records them in its constant cache, so that intern_constants() never has to rebuild a compiler-produced frozenset.

A specific reproducer for this case:

Details

"""marshal output for a module depends on which *other* code objects happen to
be alive in the interpreter when it is compiled (CPython default build).

Run:  python3 repro.py
"""
import gc
import marshal
import sys

# The module under test: the same set-literal membership test in two functions
# whose constants otherwise differ.  The compiler folds each `in {...}` into a
# frozenset constant; its per-compilation constant cache then makes f and g
# share ONE frozenset object.  (f and g must differ in some other constant;
# two functions with identical co_consts share the whole tuple, which happens
# to mask the effect.)
SUBJECT = '''
def f(kind):
    if kind in {"alpha_kind", "beta_kind"}:
        return "yes: f"

def g(kind):
    if kind in {"alpha_kind", "beta_kind"}:
        return "yes: g"
'''

# Any unrelated module that merely uses the same identifier-like strings.
# (Spelled via str.join so that *this* file's own code object does not itself
# hold the two strings as interned constants -- that alone triggers case 3.)
BYSTANDER = '''
def h(kind):
    return kind == "ALPHA" or kind == "BETA"
'''.replace("ALPHA", "_".join(["alpha", "kind"])).replace("BETA", "_".join(["beta", "kind"]))


def compile_subject(label):
    code = compile(SUBJECT, "subject.py", "exec", dont_inherit=True)
    data = marshal.dumps(code)          # what py_compile / importlib write into a .pyc
    f, g = (c for c in code.co_consts if hasattr(c, "co_code"))
    shared = (next(c for c in f.co_consts if isinstance(c, frozenset))
              is next(c for c in g.co_consts if isinstance(c, frozenset)))
    print(f"{label:44s} len(marshal.dumps(code))={len(data)}  "
          f"f and g share one frozenset object: {shared}")
    return data


print(sys.version)
a = compile_subject("1. fresh interpreter")
b = compile_subject("2. same again")
bystander = compile(BYSTANDER, "bystander.py", "exec", dont_inherit=True)
c = compile_subject("3. while bystander's code object is alive")
del bystander
gc.collect()
d = compile_subject("4. after bystander's code object is freed")

print()
print(f"identical bytes?  1==2: {a == b}   1==3: {a == c}   1==4: {a == d}")
for name, other in (("2", b), ("3", c)):
    if a != other:
        i = next(i for i, (x, y) in enumerate(zip(a, other)) if x != y)
        print(f"1 vs {name}: first difference at offset {i}: {a[i]:#04x} vs {other[i]:#04x} "
              f"(marshal type codes {chr(a[i] & 0x7f)!r} vs {chr(other[i] & 0x7f)!r}; "
              f"FLAG_REF {bool(a[i] & 0x80)} vs {bool(other[i] & 0x80)}); "
              f"code objects compare equal: {marshal.loads(a) == marshal.loads(other)}")

confirmed on 3.13, 3.14, and 3.15.

While I believe this is fixable, post-processing .pyc files as noted above, or writing a standalone compiler might make more sense given how hard it is to maintain an idempotent compile within the live state machine of a running CPython runtime.

marshal should check whether the same object is serialized more than once in a single operation, rather than whether its reference count is 1.
compiler should run intern frozenset when creating it, so that subsequent calls to intern_constants() leave them unchanged.

A smaller trigger than compile order, in the default (GIL) build: one interned string that equals an item of a frozenset constant is enough. This reproduces on 3.12.15, 3.13.15 and 3.14.8. As far as I can tell, gh-156862 only changes free-threaded builds.

import subprocess, sys

SRC = '''
def f():
    assert set(U) == {"ga", "gb", "gc", "hd", "he"}
    assert set(U) == {"g0", "g1", "g2", "h3", "h4"}
def g():
    assert set(U) == {"g0", "g1", "g2", "h3", "h4"}
'''

CHILD = r'''
import marshal, sys
keep = [sys.intern("g" + str(0))] if sys.argv[1] == "pre-interned" else []
code = compile(%r, "m.py", "exec")
f, g = code.co_consts[0], code.co_consts[1]
fs_f = [k for k in f.co_consts if isinstance(k, frozenset)]
fs_g = [k for k in g.co_consts if isinstance(k, frozenset)]
shared = any(a is b for a in fs_f for b in fs_g)
sys.stdout.write(f"{len(marshal.dumps(code))} bytes, f/g share the frozenset: {shared}")
''' % SRC

for mode in ("fresh", "pre-interned"):
    out = subprocess.run([sys.executable, "-c", CHILD, mode], capture_output=True, text=True)
    print(f"{mode:13} {out.stdout or out.stderr}")

3.12.15  fresh         517 bytes, f/g share the frozenset: True
3.12.15  pre-interned  542 bytes, f/g share the frozenset: False
3.13.15  fresh         523 bytes, f/g share the frozenset: True
3.13.15  pre-interned  548 bytes, f/g share the frozenset: False
3.14.8   fresh         545 bytes, f/g share the frozenset: True
3.14.8   pre-interned  570 bytes, f/g share the frozenset: False

marshal.loads() of the two results compares equal; only the encoding differs. With the frozenset shared, the second occurrence is a TYPE_REF. Without sharing, it is written out in full (+25 bytes here), and every later ref index shifts by one.

What seems to happen: the compiler merges the two equal frozensets into one object. When the code objects are created, intern_constants() (Objects/codeobject.c) turns each frozenset into a tuple and interns its strings. If that replaces an item with an already-interned equal string (tmp_modified), it builds a new frozenset for that code object. So f and g each get their own copy, and the sharing that marshal relies on is gone. Whether the replacement happens depends on what the process interned before, for example identifiers of previously compiled modules. That is why compileall -j N and uv's worker pool produce different .pyc from run to run. I have not worked out why the extra frozenset in f is needed to trigger it.

Real-world hit: networkx/algorithms/operators/tests/test_binary.py and pycparser/c_parser.py change in about 1–2 of 5 Docker builds of the same lockfile (uv, 3.13), so the venv layer of the image changes.

pythongh-129724: Make marshal output independent of reference counts

pythongh-159184: Make the marshal order of set elements with equal du…

pythongh-129724: Make marshal output independent of reference counts

pythongh-159184: Make the marshal order of set elements with equal du…

eyeballs on the pair of PRs would be helpful here.

Merge the pythongh-156504 NEWS entry into the pythongh-129724 one

gh-129724: gh-156504: Make marshal output independent of reference co…

gh-129724: Intern string constants in the compiler, not per code obje…

pythongh-159184: Make the marshal order of set elements with equal du…

Navigation Menu

Repository navigation

Bytecode compilation output depends on order of files compiled #129724

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Bytecode compilation output depends on order of files compiled #129724

Description

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

encukou commented on Feb 6, 2025

konstin commented on Feb 6, 2025

encukou commented on Feb 6, 2025

gpshead commented on Oct 4, 2026

methane commented on Oct 4, 2026

tran4774 commented on Oct 8, 2026

gpshead commented on Oct 11, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions