support reproducible Python builds

@warsaw

BPO	29708
Nosy	@warsaw, @vstinner, @ericvsmith, @benjaminp, @mcepl, @merwok, @methane, @zooba, @dstufft, @bmwiedemann, @FRidh, @commodo, @mingwandroid, @eli-schwartz, @miss-islington, @jefferyto, @obfusk
PRs	bpo-29708: support SOURCE_DATE_EPOCH env var in py_compile (allow for reproducible builds of python packages) #296 bpo-29708: allow to force hash-based pycs #5200 bpo-29708: Add What's New entries for SOURCE_DATE_EPOCH and py_compile #5306 bpo-29708: support SOURCE_DATE_EPOCH for build info #5313 bpo-34033: distutils: byte_compile() sort files #8057 bpo-34022: Stop forcing of hash-based invalidation with SOURCE_DATE_EPOCH #9607 [3.7] bpo-34022: Stop forcing of hash-based invalidation with SOURCE_DATE_EPOCH (GH-9607) #10775
Files	python39_2.html: Python 3.9.1 diffoscope report

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-03-03.11:36:06.114>
labels = ['build', '3.10']
title = 'support reproducible Python builds'
updated_at = <Date 2021-04-22.17:01:17.438>
user = 'https://github.com/bmwiedemann'

bugs.python.org fields:

activity = <Date 2021-04-22.17:01:17.438>
actor = 'obfusk'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Build']
creation = <Date 2017-03-03.11:36:06.114>
creator = 'bmwiedemann'
dependencies = []
files = ['49708']
hgrepos = []
issue_num = 29708
keywords = ['patch']
message_count = 40.0
messages = ['288880', '288883', '288889', '288948', '301354', '309394', '309395', '309401', '309870', '309905', '309931', '309972', '310010', '310012', '310292', '310636', '310637', '310652', '310661', '311317', '313312', '313313', '313383', '313384', '313391', '320942', '320989', '321002', '327480', '330623', '347971', '384065', '384066', '384099', '384100', '384104', '384110', '384123', '386272', '391616']
nosy_count = 18.0
nosy_names = ['barry', 'vstinner', 'eric.smith', 'benjamin.peterson', 'mcepl', 'eric.araujo', 'sascha_silbe', 'methane', 'steve.dower', 'dstufft', 'bmwiedemann', 'Frederik Rietdijk', 'Alexandru Ardelean', 'Ray Donnelly', 'eschwartz', 'miss-islington', 'jefferyto', 'obfusk']
pr_nums = ['296', '5200', '5306', '5313', '8057', '9607', '10775']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue29708'
versions = ['Python 3.10']

See https://reproducible-builds.org/ and https://reproducible-builds.org/docs/buy-in/ for why this is a good thing to have in general.

Fedora, openSUSE and possibly other Linux distributions package .pyc files as part of their binary rpm packages and they are not trivial to drop [1].

A .pyc header includes the timestamp of the source .py file
which creates non-reproducible builds when the .py file is touched during build time (e.g. for a version.py).
As of 2017-02-10 in openSUSE Factory this affected 476 packages (such as python-amqp and python3-Twisted).

[1] http://lists.opensuse.org/opensuse-packaging/2017-02/msg00086.html

--
Eric.

On Mar 3, 2017, at 6:36 AM, Bernhard M. Wiedemann <report@bugs.python.org> wrote:

New submission from Bernhard M. Wiedemann:

See https://reproducible-builds.org/ and https://reproducible-builds.org/docs/buy-in/ for why this is a good thing to have in general.

Fedora, openSUSE and possibly other Linux distributions package .pyc files as part of their binary rpm packages and they are not trivial to drop [1].

A .pyc header includes the timestamp of the source .py file
which creates non-reproducible builds when the .py file is touched during build time (e.g. for a version.py).
As of 2017-02-10 in openSUSE Factory this affected 476 packages (such as python-amqp and python3-Twisted).

[1] http://lists.opensuse.org/opensuse-packaging/2017-02/msg00086.html

----------
components: Build, Distutils
messages: 288880
nosy: bmwiedemann, dstufft, merwok
priority: normal
pull_requests: 353
severity: normal
status: open
title: support reproducible Python builds
versions: Python 2.7, Python 3.3, Python 3.4, Python 3.5, Python 3.6

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue29708\>

New-bugs-announce mailing list
New-bugs-announce@python.org
https://mail.python.org/mailman/listinfo/new-bugs-announce

Shouldn't this at least also cover Python 3.7? And should it be officially backported? I would think that if #296 gets accepted for 3.7, then distros that care can cherry pick it back into whatever versions they still support. It probably needn't be officially cherry picked upstream.

(FWIW, this doesn't affect the Debian ecosystem since we don't ship pycs in debs.)

backports are optional.
It can help reduce duplicated work for the various distributions.
Currently, I think master and 2.7 are the most relevant targets.

I have proposed PEP-552 to address this issue.

Hey,

Allow me to join the discussion here.

Context:

I'm the maintainer of Python & Python3 in the OpenWrt distro, and (since a while) we also care about reproducible builds.
The person [Alexander Couzens] who's leading the effort for OpenWrt, has pinged me about Python(3) and packages [to see about making them reproducible]
In OpenWrt we *only* ship .pyc files, because of performance considerations [.pyc can be 10x faster than .py on some SoCs], and size limitation [we cannot allow auto .pyc generation since it can be expensive on RAM [ < 32 MB systems ] or flash [ ~8 MB sizes ] ; believe it or not, people run Python on something like this

Current status:

so far I've implemented a simple change to Python & Python3 here:
openwrt/packages@1b6dd47
that has improved reproduce-ability quite a bit : only binaries are not reproduce-able now
when I did this [1-2 weeks ago] I did not think of checking of any bug/issue opened here [ I only thought if this now ]
I only checked what other distros may do regarding Python:
https://tests.reproducible-builds.org/debian/reproducible.html

References:

initial discussion on OpenWrt: [discussion] should we remove .pyc files from all packages? openwrt/packages#5278
PR with discussion: python,python3: add support for SOURCE_DATE_EPOCH var openwrt/packages#5303
current OpenWrt reproducible state [with the patch applied]: https://tests.reproducible-builds.org/lede/lede_ar71xx.html

I wanted to share my [and our] interest in this.

If we can help in any way, feel free to ping.

I will try to hack/patch some more stuff in the current Python releases to make them fully reproducible [for us], and probably share the results here.
When PEP-552 gets implemented and there will be a Python we will switch to them.
Atm, in trunk we package Python 2.7.14 & Python 3.6.4

Thanks
Alex

PEP-552 has been implemented for 3.7.

Thank you for the heads-up.
I did not follow-up too in-depth on the resolution.

I just stumbled over this last night.

Will keep an eye for 3.7, and see about 2.7.

A disagreement has popped up over what the ideal solution is on the PR currently connected to this issue. I'm having the folks involved switch it over to here.

IMO I think py_compile can respect SOURCE_DATE_EPOCH and just blindly use it for creating .pyc files. That way builds are reproducible. Yes, it will quite possibly lead to those .pyc files being regenerated the instant Python starts running, but SOURCE_DATE_EPOCH is entirely about builds, not runtimes. Plus .pyc files are just optimizations and so it is not critical they not be regenerated again later.

@bmwiedemann

So, a couple of things.

It seems to me, that properly supporting SOURCE_DATE_EPOCH means using exactly that and nothing else. To that end, I'm not entirely sure why things like --clamp-mtime even exist, as the original timestamp of a source file doesn't seem to have a lot of utility and it is better to be entirely predictable. But I'm not going to argue that, except insomuch as it seems IMHO to fit better for python to just keep things simple and override the timestamp with the value of SOURCE_DATE_EPOCH

That being said, I see two problems with python implementing something analogous to --clamp-mtime rather than just --mtime.

Source files are extracted by some build process, and remain untouched. Python generates bytecode pinned to the original time, rather than SOURCE_DATE_EPOCH. Later, the build process packages those files and implements --mtime, not --clamp-mtime. Because Python and the packaging software disagree about which one to use, the bytecode fails.
Source files are extracted, and the build process even tosses all timestamps to the side of the road, by explicitly touching all of them to the date of SOURCE_DATE_EPOCH just in case. Then for whatever reason (distro patches, 2to3, the use of cp) the timestamps get updated to $currentime. But SOURCE_DATE_EPOCH is in the future, so the timestamps get downdated. Python bytecode is generated by emulating --clamp-mtime. The build process then uses --mtime to package the files. Again, because Python and the packaging software disagree about which one to use, the bytecode fails.

Of course, in both those cases, blindly respecting SOURCE_DATE_EPOCH will seemingly break everything for people who use --clamp-mtime instead. I'm not happy with reproducible-builds.org for allowing either one.

I don't think python should rely on --mtime users manually overriding the filesystem metadata of the source files outside of py_compile, as that is a hack that I think we'd like to remove if possible... that being said, Arch Linux will, on second thought, not be adversely affected even if py_compile tries to be clever and emulate --clamp-mtime to decide on its own whether to respect SOURCE_DATE_EPOCH.

Likewise, I don't really expect people to try to reproduce builds using a future date for SOURCE_DATE_EPOCH. On the other hand, the reproducible builds spec doesn't forbid it AFAICT.

But... neither of those mitigations seem "clean" to me, for the reasons stated above.

There is something that would solve all these issues, though. From reading the importlib code (I haven't actually tried smoketesting actual imports), it appears that Python 2 accepts any bytecode that is dated at or later than the timestamp of its source .py, while Python 3 requires the timestamps to perfectly match. This seems bizarre to behave differently, especially as until @bmwiedemann mentioned it on the GitHub PR I blindly assumed that Python would not care if your bytecode is somehow dated later than your sources. If the user is playing monkey games with mismatched source and byte code, while backdating the source code to *trick* the interpreter into loading it... let them? They can break their stuff if they want to!

On looking through the commit logs, it seems that Python 3 used to do the same, until 61b1425 refactored the general vicinity and modified this behavior without warning. In a commit that seems to be designed to do something else entirely. This really should have been two separate commits, and modifying the import code to more strictly check the timestamp should have come with an explanatory justification. Because I cannot think of a good reason for this behavior, and the commit isn't giving me an opportunity to understand either. As it is, I am completely confused, and have no idea whether this was even supposed to be deliberate.
In hindsight it is certainly preventing nice solutions to supporting SOURCE_DATE_EPOCH.

@bmwiedemann

As Eli's comments are coming off as negative to/at me, I feel like I have
to defend myself here. If you look at the commit there was actually two
places where the timestamp was checked; one did an equality comparison and
one did a >= comparison. It's quite possible the semantics accidentally
changed as part of the refactoring due to the check being done in different
places and a different one was copied, although no one has even noticed
until now.

If there is a desire to change the semantics of how timestamps are checked
then that should be done in a separate issue as at this point we have lived
with the current semantics for several releases -- all releases of Python 3
still receiving security updates -- so it's passed being a bug and is now
the semantics in Python 3.

On Sat, Jan 13, 2018, 16:57 Eli Schwartz, <report@bugs.python.org> wrote:

Eli Schwartz eschwartz93@gmail.com added the comment:

So, a couple of things.

It seems to me, that properly supporting SOURCE_DATE_EPOCH means using
exactly that and nothing else. To that end, I'm not entirely sure why
things like --clamp-mtime even exist, as the original timestamp of a source
file doesn't seem to have a lot of utility and it is better to be entirely
predictable. But I'm not going to argue that, except insomuch as it seems
IMHO to fit better for python to just keep things simple and override the
timestamp with the value of SOURCE_DATE_EPOCH

That being said, I see two problems with python implementing something
analogous to --clamp-mtime rather than just --mtime.

Source files are extracted by some build process, and remain untouched.
Python generates bytecode pinned to the original time, rather than
SOURCE_DATE_EPOCH. Later, the build process packages those files and
implements --mtime, not --clamp-mtime. Because Python and the packaging
software disagree about which one to use, the bytecode fails.

Source files are extracted, and the build process even tosses all
timestamps to the side of the road, by explicitly touching all of them to
the date of SOURCE_DATE_EPOCH just in case. Then for whatever reason
(distro patches, 2to3, the use of cp) the timestamps get updated to
$currentime. But SOURCE_DATE_EPOCH is in the future, so the timestamps get
downdated. Python bytecode is generated by emulating --clamp-mtime. The
build process then uses --mtime to package the files. Again, because Python
and the packaging software disagree about which one to use, the bytecode
fails.

Of course, in both those cases, blindly respecting SOURCE_DATE_EPOCH will
seemingly break everything for people who use --clamp-mtime instead. I'm
not happy with reproducible-builds.org for allowing either one.

I don't think python should rely on --mtime users manually overriding the
filesystem metadata of the source files outside of py_compile, as that is a
hack that I think we'd like to remove if possible... that being said, Arch
Linux will, on second thought, not be adversely affected even if py_compile
tries to be clever and emulate --clamp-mtime to decide on its own whether
to respect SOURCE_DATE_EPOCH.

Likewise, I don't really expect people to try to reproduce builds using a
future date for SOURCE_DATE_EPOCH. On the other hand, the reproducible
builds spec doesn't forbid it AFAICT.

But... neither of those mitigations seem "clean" to me, for the reasons
stated above.

There is something that would solve all these issues, though. From reading
the importlib code (I haven't actually tried smoketesting actual imports),
it appears that Python 2 accepts any bytecode that is dated at or later
than the timestamp of its source .py, while Python 3 requires the
timestamps to perfectly match. This seems bizarre to behave differently,
especially as until @bmwiedemann mentioned it on the GitHub PR I blindly
assumed that Python would not care if your bytecode is somehow dated later
than your sources. If the user is playing monkey games with mismatched
source and byte code, while backdating the source code to trick the
interpreter into loading it... let them? They can break their stuff if they
want to!

On looking through the commit logs, it seems that Python 3 used to do the
same, until
61b1425
refactored the general vicinity and modified this behavior without warning.
In a commit that seems to be designed to do something else entirely. This
really should have been two separate commits, and modifying the import code
to more strictly check the timestamp should have come with an explanatory
justification. Because I cannot think of a good reason for this behavior,
and the commit isn't giving me an opportunity to understand either. As it
is, I am completely confused, and have no idea whether this was even
supposed to be deliberate.
In hindsight it is certainly preventing nice solutions to supporting
SOURCE_DATE_EPOCH.

----------
nosy: +eschwartz

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue29708\>

I think, there is no single nice and clean solution with time-based .pyc files, but to get a whole distribution to build reproducibly, there are two other ways:

if the SOURCE_DATE_EPOCH environment variable is set,
make hash-based .pyc files the default.
instead of storing .py mtime in the .pyc header, use the .pyc's filesystem mtime value - also making it more available to users.
Not sure if this would have side-effects or cause regressions.

on the side-issue: IMHO checking exact mtimes is the right thing to do, because sometimes users will copy back old .py files and expect mismatching .pyc files to not be used.

What if the entire python standard libraries was to be rewritten in C?

This is such an enormous task that it will never happen. Plus it would make CPython harder to maintain, and make it harder to share anything in the stdlib with other implementations.

What if the entire python standard libraries was to be rewritten in C

Lots of pointer management boilerplate, plus Python operators will inflate into lenghty function calls.

Alternatively python could build in cython into it’s compile() function under python to compile scripts into pyd’s. However the generated c code would have to be deleted to not waste a ton of disk space.

Using Cython as an optional feature enabled in configure and PCBuild looks promising.

However, we need a person who is ready to add the lines into configure.ac, Makefile.pre.in, PCBuild/get_externals.bat, relevant Visual Studio projects, and possibly submit necessary patches to Cython to support such a use case.

If compiled with PGO twice, it will also produce inconsistencies, and there are many different places.

What's the reason for this

If compiled with PGO twice, it will also produce inconsistencies, and there are many different places.

I don't understand what you are saying.
Please provide concrete and complete step to reproduce.

If compiled with PGO twice, it will also produce inconsistencies, and there are many different places.

I don't understand what you are saying. Please provide concrete and complete step to reproduce.

I don’t understand why #93317 didn’t get linked here.

It was, but the UI is not very obvious: #73894 (reference)

I am a novice on python compilation and trying determininstic pyc. As simplest use case, I cloned git repo with a sample .py file in two places (src1 and src2) inside the same build environment (docker container) and compiled both of them using invalidation_mode=CHECKED_HASH and SOURCE_DATE_EPOCH set. But I can see that hash of corresponding .pyc files are different. (source file inside both src1 and src2 have same hash but different file attributes notably mtime)

How can I compile source file in src1 and src2 so that correponding pyc have same hash? I am using python version 3.10.12. My apologies if this is not the right place to ask.

On Mon Feb 26, 2024 at 5:58 PM CET, compete2cooperate wrote:

How can I compile source file in src1 and src2 so that correponding pyc have same hash? I am using python version 3.10.12. My apologies if this is not the right place to ask.

Look at patches we, openSUSE, have in our packages. Particularly relevant are I believe distutils-reproducible-compile.patch, gh-78214-marshal_stabilize_FLAG_REF.patch, and bpo-37596-make-set-marshalling.patch. Perhaps also python-3.3.0b1-fix_date_time_compiler.patch (I am not sure whether that one is not actually obsolete).

It seems like the main issue has been solved. Would it make sense to open separate issues for the remaining specific problems and close this one?

Yes, let's do that instead of having one big issue that is quite old.

pythongh-73894: Don't let GC timing decide FLAG_REF in the free-threa…

Navigation Menu

Repository navigation

support reproducible Python builds #73894

46 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

support reproducible Python builds #73894

Description

Activity

bmwiedemann commented on Mar 3, 2017

ericvsmith commented on Mar 3, 2017

warsaw commented on Mar 3, 2017

bmwiedemann commented on Mar 4, 2017

benjaminp commented on Sep 5, 2017

commodo commented on Jan 2, 2018

benjaminp commented on Jan 2, 2018

commodo commented on Jan 3, 2018

brettcannon commented on Jan 12, 2018

eli-schwartz commented on Jan 14, 2018

brettcannon commented on Jan 14, 2018

bmwiedemann commented on Jan 15, 2018

46 remaining items

ericvsmith commented on Jun 12, 2022

arhadthedev commented on Jun 12, 2022

arhadthedev commented on Jun 12, 2022

zhuofeng6 commented on Jun 17, 2022

methane commented on Jun 17, 2022

mcepl commented on Aug 1, 2022

merwok commented on Aug 1, 2022

compete2cooperate commented on Feb 26, 2024

mcepl commented on Feb 26, 2024

manueljacob commented on May 9, 2026

picnixz commented on May 9, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions