Repository navigation
Running the Python test suite leaks .pem files in /tmp #93353
Description
Activity
The problem comes from pip bootstrap when running the test_with_pip() test of test_venv.
It seems like pip creates a .pem file in the temporary directory (like /tmp), but only when pip is imported from a ZIP file, which is the case when I run get-pip.py: https://bootstrap.pypa.io/get-pip.py
I can reproduced the issue in the 3.9, 3.10, 3.11 and main branches (I didn't test older branches).
- In Python 3.9, I cannot reproduce the issue in Python 3.9 at commit 4b4d60f: pip 20.2.3.
- But I can reproduce the issue at commit d962b00: pip 21.1.
It seems to be a change between pip 20.2.3 and pip 21.1.
In get-pip.py,
from pip._internal.commands.install import InstallCommandis enough to create the.pemfile.Shell script to reproduce the issue:
set -e -x DIR=$PWD/TMP/ rm -rf $DIR mkdir $DIR rm -rf env TMPDIR=$DIR TEMPDIR=$DIR ./python -m venv env --without-pip TMPDIR=$DIR TEMPDIR=$DIR env/bin/python -m ensurepip -v echo echo echo "=== TMPDIR ===" find $DIR
Output:
(...) === TMPDIR === ++ find /home/vstinner/python/main/TMP/ /home/vstinner/python/main/TMP/ /home/vstinner/python/main/TMP/tmpmb5ndxc7cacert.pemThe PEM file is created indirectly by importlib.resources on
import pip._vendor.requests.It's created by this code:
>>> import sys; sys.path.insert(0, 'pip.zip') >>> import pip._vendor.certifi >>> pip._vendor.certifi.where() pip.zip/pip/_vendor/certifi/core.py:50: DeprecationWarning: path is deprecated. Use files() instead. Refer to https://importlib-resources.readthedocs.io/en/latest/using.html#migrating-from-legacy for migration advice. '/home/vstinner/python/main/TMP/tmp9osgb6wrcacert.pem'
Debugger:
Lib/importlib/resources/_common.py, line 84, in _tempfile (suffix='cacert.pem', fd=4, raw_path='/home/vstinner/python/main/TMP/tmpv7fwzaoocacert.pem')If Python exits normally, the garbage collector deletes the temporary PEM file. When running ensurepip / get-pip.py, it's not deleted. I don't know why.
importlib.resources only creates a temporary file if pip is imported from a ZIP file.
Workaround to manually delete the temporaryfile:
pip._vendor.certifi.core._CACERT_CTX = None
OpenSSL requires a filename to a PEM file. certifi uses importlib.resources to create a temporary named file if pip._vendor.certifi is imported from a ZIP file. That's convenient.
The problem is more that the
pip._vendor.certifi.core._CACERT_CTXcontext manager which keeps the temporary file alive until it's destroyed. Maybe sometimes_CACERT_CTXis part of a complex reference cycle, it's never destroyed, and so the temporary file is not deleted.Nice. How did you find this?
I used a shell script similar to #93353 (comment) and I ran manually a bisection on the test list (./python -m test --list-tests) using a script taking a random.sample() of half of tests. I repeated the operation until I found a single test file, then I looked at the code.
The leaked files was reported by more and more owners of buildbot workers.
Once the PEM issue is solved, my plan is to enhance test.regrtest to do something similar than the shell script: define TMPFILE env var. Maybe in the main process spawning test processes.
For pip/certifi, I hacked get-pip.py to add breakpoint() and I followed the code flow. It's not easy to identify which part of pip created the file. It's not just an import. It's an import + call to where() function, done in the module body that I failed to find which module. Maybe requests. It would be nice to be able to postpone when where() is called to workaround the issue: only call it when a filename to cacert.pem is needed.
Reproducer without pip nor certifi, attached run.py and bug.py scripts:
$ ./python run.py bug.py: temporary directory: ['tmp1wkab03ycacert.pem'] run.py: temporary directory: ['tmp1wkab03ycacert.pem']The expected output is an empty list in run.py (parent process). bug.py never calls _tempfile() finally block.
I'm still bisecting the issue. It smells like logging prevents deleting an object somehow.
Ok, this problem is quite complicated:
- pip imports requests
- requests imports certifi
- requests calls certifi.where()
- certifi.where() uses importlib.resources to create a temporary file using a context manager.
- logging is imported by pip
- logging calls os.register_at_fork() which keeps the whole logging namespace alive
- somehow, the logging module keeps the whole pip package (!) and all of its sub-modules (!) alive until Python clears os.register_at_fork(): that happens VERY LATE, like the last function call before the last GC collection.
- Python clears os.register_at_fork() callbacks: the context manager
finally:block is called as expected.... - Oops,
os.removeisNoneat this point, the call fails.
Fixing importlib.resources is simple: keep a reference to os.remove() in importlib.resources implementation to make sure that we can remove the file very late during Python finalization.
This problem is tricky because:
- importlib.resources only creates a named temporary file if pip is imported from a ZIP file. When pip is installed on disk as a file, the bug is gone.
- the code is valid.
- certifi relies on the garbage collector to gently calls the context manager finalizer which removes the temporary file.
- Python doesn't log any error when a bug occurs very late during Python finalization.
I'm working on a fix for importlib.resources.
12 remaining items
- added a commit that references this issue
on Jun 14, 2022 PR GH-93776 broke testing on WASI:
_PYTHON_HOSTRUNNER='wasmtime run --env PYTHONPATH=/build_oot/host/build/lib.wasi-wasm32-3.12:/Lib --mapdir /::../.. --' _PYTHON_PROJECT_BASE=/buildmaster-config/master/workers/local-worker/3.x.local-worker.wasi/build/build_oot/host _PYTHON_HOST_PLATFORM=wasi-wasm32 PYTHONPATH=/buildmaster-config/master/workers/local-worker/3.x.local-worker.wasi/build/build_oot/host/build/lib.wasi-wasm32-3.12:../../Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_d_wasi_wasm32-wasi ../build/python ../../Tools/scripts/run_tests.py -j 1 -u all -W --slowest --fail-env-changed --timeout=900 -j2 --junit-xml test-results.xml == CPython 3.12.0a0 (heads/main-dirty:a338e106b6, Jun 14 2022, 13:20:18) [GCC 9.4.0] == Linux-5.17.13-300.fc36.x86_64-x86_64-with-glibc2.31 little-endian == cwd: /buildmaster-config/master/workers/local-worker/3.x.local-worker.wasi/build/build_oot/host/build/test_python_111802æ == CPU count: 8 == encodings: locale=UTF-8, FS=utf-8 Using random seed 6754872 0:00:00 load avg: 5.20 Run tests in parallel using 2 child processes (timeout: 15 min, worker timeout: 20 min) Warning -- regrtest worker thread failed: Traceback (most recent call last): Warning -- File "/buildmaster-config/master/workers/local-worker/3.x.local-worker.wasi/build/Lib/test/libregrtest/runtest_mp.py", line 325, in run Warning -- mp_result = self._runtest(test_name) Warning -- ^^^^^^^^^^^^^^^^^^^^^^^^ Warning -- File "/buildmaster-config/master/workers/local-worker/3.x.local-worker.wasi/build/Lib/test/libregrtest/runtest_mp.py", line 280, in _runtest Warning -- os.mkdir(tmp_dir) Warning -- ^^^^^^^^^^^^^^^^^ Warning -- FileExistsError: [Errno 17] File exists: '/buildmaster-config/master/workers/local-worker/3.x.local-worker.wasi/build/build_oot/host/build/test_python_111802æ_tmpdir' Kill <TestWorkerProcess #1 running test=test_typing pid=111806 time=2 ms> process groupThe commit e566ce5 relies on an environment variable to create unique temp directories per instance. The approach does not work on WASI for two reasons:
- WASI runtimes do not pass environment variables to WASM programs.
python.wasmprocess does not see theTMPDIRvariable. - wasmtime uses an explicit, sandboxed mapping of host directories to runtime directories (sort of a chroot, but better). The
TMPDIRoutside would not be the same as theTMPDIRinside.
- WASI runtimes do not pass environment variables to WASM programs.
- added a commit that references this issue
on Jun 14, 2022 The initial issue (leaked PEM files) has been fixed.
regrtest now detects leaked temporary files in the main branch. I don't want to backport this change since it causes some CI issues, and I'm still fixing regrtest. I have a few more changes to enhance regrtest which will use this issue number, but IMO the issue can be closed since the initial issue has been fixed.
- added a commit that references this issue
on Jun 16, 2022 - added a commit that references this issue
on Jun 26, 2022 - added 4 commits that reference this issue
on Sep 2, 2023
Running the Python test suite has two issues:
Moreover, test_tools enters an unlimited loop and fills the $TMPDIR directory if the $TMPDIR is a sub-directory of the Python source code directory. Example:
Running test_freeze_simple_script() of test_tools copies TMP/ into TMP/TMP/ and then into TMP/TMP/TMP/, etc. Quickly, it fills TMP/ with a "loop" of files :-)
Linked PRs