Repository navigation
Scripts/*.exe files are not reproducible (due to embedded timestamp) #164
Description
*.dist-info/RECORD files are not reproducible on Windows for packages that define entry_points/console_scripts that go into the Scripts/ folder when installed by pip.
For example, after running pip install wheel or pip install pyinstaller,
in wheel-0.37.1.dist-info/RECORD, I have this line:
../../Scripts/wheel.exe,sha256=u9TbPw2XNs_F9uy7y2zwumuzAZDbOSB7BXjLHZ0tTHg,97103
in pyinstaller-4.10.dist-info/RECORD, I have these lines:
../../Scripts/pyi-archive_viewer.exe,sha256=nC-9muPlIhUC1qvFkXHpyKJyRQqXISXxbUPXQ1XVOiM,97133
../../Scripts/pyi-bindepend.exe,sha256=udFHiAdndPpSwaIqmhmLEy36IUs1cNNoNQznSEnLJQQ,97128
../../Scripts/pyi-grab_version.exe,sha256=3ET9E841tFWujFL99aG4frzgwlP9f9pAkMgE0k2UGK0,97131
../../Scripts/pyi-makespec.exe,sha256=dJkfmITdLJhyPngmqziqqj5tH9qqfeQc5BTubeoXWUs,97127
../../Scripts/pyi-set_version.exe,sha256=sWmcOVS93fUY-wbdoz6ixBCvjy1tC4Aaw30DMmrmo-0,97130
../../Scripts/pyinstaller.exe,sha256=haInbhH0pImJn24cW4v917oUZmzXZj8OE89KFh4MO2Y,97112
These exe files and hence the RECORD files themselves are not reproducible.
Upon comparing multiple Scripts/wheel.exe files, I've found that the only difference is due to a timestamp embedded inside the exe (or rather, same timestamp embedded twice).
The exe files get created by distlib (vendored by pip).
Here is a traceback with an artificial exception to illustrate the codepath:
(env) PS C:\tmp> pip install --no-build-isolation pyinstaller
Collecting pyinstaller
Using cached pyinstaller-4.10-py3-none-win_amd64.whl (2.0 MB)
Requirement already satisfied: setuptools in c:\tmp\env\lib\site-packages (from pyinstaller) (61.0.0)
Requirement already satisfied: pyinstaller-hooks-contrib>=2020.6 in c:\tmp\env\lib\site-packages (from pyinstaller) (2022.3)
Requirement already satisfied: altgraph in c:\tmp\env\lib\site-packages (from pyinstaller) (0.17.2)
Requirement already satisfied: pefile>=2017.8.1 in c:\tmp\env\lib\site-packages (from pyinstaller) (2021.9.3)
Requirement already satisfied: pywin32-ctypes>=0.2.0 in c:\tmp\env\lib\site-packages (from pyinstaller) (0.2.0)
Requirement already satisfied: future in c:\tmp\env\lib\site-packages (from pefile>=2017.8.1->pyinstaller) (0.18.2)
Installing collected packages: pyinstaller
ERROR: Exception:
Traceback (most recent call last):
File "C:\tmp\env\lib\site-packages\pip\_internal\cli\base_command.py", line 167, in exc_logging_wrapper
status = run_func(*args)
File "C:\tmp\env\lib\site-packages\pip\_internal\cli\req_command.py", line 205, in wrapper
return func(self, options, args)
File "C:\tmp\env\lib\site-packages\pip\_internal\commands\install.py", line 405, in run
installed = install_given_reqs(
File "C:\tmp\env\lib\site-packages\pip\_internal\req\__init__.py", line 73, in install_given_reqs
requirement.install(
File "C:\tmp\env\lib\site-packages\pip\_internal\req\req_install.py", line 769, in install
install_wheel(
File "C:\tmp\env\lib\site-packages\pip\_internal\operations\install\wheel.py", line 729, in install_wheel
_install_wheel(
File "C:\tmp\env\lib\site-packages\pip\_internal\operations\install\wheel.py", line 646, in _install_wheel
generated_console_scripts = maker.make_multiple(scripts_to_generate)
File "C:\tmp\env\lib\site-packages\pip\_vendor\distlib\scripts.py", line 440, in make_multiple
filenames.extend(self.make(specification, options))
File "C:\tmp\env\lib\site-packages\pip\_internal\operations\install\wheel.py", line 427, in make
return super().make(specification, options)
File "C:\tmp\env\lib\site-packages\pip\_vendor\distlib\scripts.py", line 429, in make
self._make_script(entry, filenames, options=options)
File "C:\tmp\env\lib\site-packages\pip\_vendor\distlib\scripts.py", line 329, in _make_script
self._write_script(scriptnames, shebang, script, filenames, ext)
File "C:\tmp\env\lib\site-packages\pip\_vendor\distlib\scripts.py", line 263, in _write_script
raise Exception(f"heyheyhey2. {sha256(launcher)=}. {sha256(shebang)=}. {sha256(zip_data)=}. " +
Exception: heyheyhey2. sha256(launcher)='a00a877acefc'. sha256(shebang)='58628e924f22'. sha256(zip_data)='a423496a0482'. ('SOURCE_DATE_EPOCH' in os.environ)=True
The interesting code is here:
Lines 251 to 252 in d0e3f49
This calls into the cpython standard library, where
time.time() gets written into the file:https://github.com/python/cpython/blob/20e6e5636a06fe5e1472062918d0a302d82a71c3/Lib/zipfile.py#L1816-L1817
zinfo = ZipInfo(filename=zinfo_or_arcname,
date_time=time.localtime(time.time())[:6])Ideally, either distlib or the stdlib zipfile module should be changed to respect SOURCE_DATE_EPOCH, but it's not entirely clear to me which...
It is unfortunate that the ZipFile.writestr API exposed by the stdlib does not allow specifying the timestamp.
related spesmilo/electrum#7739
Activity
[-]`Scripts/*.exe` files are not reproducible (due to embedded ntimestamp)[/-][+]`Scripts/*.exe` files are not reproducible (due to embedded timestamp)[/+]respect SOURCE_DATE_EPOCH when creating `.exe` scripts
SomberNight commentedon Mar 25, 2022
I've also opened an issue on the python issue tracker, in case core developers think it should be fixed in the stdlib:
https://bugs.python.org/issue47123
win build: fix reproducibility by excluding `*.dist-info/`
build: update pip to version 23.2.1
build: update pip to version 23.2.1
build: update pip to version 23.2.1
build: update pip to version 23.2.1
build: update pip to version 23.2.1
build: update pip to version 23.2.1
build: update pip to version 23.2.1
build: update pip to version 23.2.1
build: update pip to version 23.2.1
build: update pip to version 23.2.1
build: update pip to version 23.2.1
build: update pip to version 23.2.1
build: Update to Python 3.11.8
win build: fix reproducibility by excluding `*.dist-info/`