Repository navigation
Support embedding the standard library in an executable #70195
Description
| BPO | 26007 |
|---|---|
| Nosy | @malemburg, @pfmoore |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2016-01-04.16:13:13.451>
labels = ['interpreter-core', 'type-feature']
title = 'Support embedding the standard library in an executable'
updated_at = <Date 2020-09-11.23:15:20.621>
user = 'https://bugs.python.org/philthompson10'bugs.python.org fields:
activity = <Date 2020-09-11.23:15:20.621>
actor = 'brett.cannon'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2016-01-04.16:13:13.451>
creator = 'philthompson10'
dependencies = []
files = []
hgrepos = []
issue_num = 26007
keywords = []
message_count = 12.0
messages = ['257464', '257465', '257466', '257469', '257470', '257477', '257480', '257493', '257495', '257496', '257503', '376748']
nosy_count = 3.0
nosy_names = ['lemburg', 'paul.moore', 'philthompson10']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'test needed'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue26007'
versions = ['Python 3.5', 'Python 3.6']
Activity
philthompson10 commentedon Jan 4, 2016
The use case is a packaging tool that can create a single executable for a Python application. Like similar tools it embeds frozen Python code (including the standard library) and is linked (often statically) against the interpreter library.
Executables are usually configured so that they cannot import packages from the filesystem. They can only import packages embedded in the executable. This is done directly, ie. the package is not written out to the filesystem and imported from there. This is done by a specially written importer.
The importer is installed by adding it to sys.path_hooks. However this must be done early on in the bootstrap process so that the parts of the standard library that are implemented as Python packages can be found by the installer. For example, the point at which the zipimport importer is added to sys.path_hooks is too late.
Currently a modified version of bootstrap_external.py is embedded in the executable which updates sys.path_hooks at the end of the _install() function.
What I would like is some way to avoid having to modify bootstrap_external.py to install a new importer sufficiently early in the bootstrap process to allow it to import the standard library.
pfmoore commentedon Jan 4, 2016
Why is it too late to install a new importer to sys.path_hooks when zipimport is installed? As far as I am aware, running the stdlib from a zipfile works fine, which seems to be equivalent to your use case.
philthompson10 commentedon Jan 4, 2016
The problem is the import of the encodings module in _PyCodecRegistry_Init().
malemburg commentedon Jan 4, 2016
So the use case is that you want to explicitly prevent the file system from being used for imports ?
You might want to look at how eGenix PyRun works, since this does embed the Python stdlib into the executable (including the encodings package). While it does allow imports from the file system, it only looks there for things it cannot find embedded in the binary.
PS: I'd suggest to make the issue title more specific to what you are actually after, which seems to be a way to install importers from C at a very early stage during initialization.
philthompson10 commentedon Jan 4, 2016
Yes, preventing the filesystem being used for imports would be another way of expressing the issue.
Regarding the title I specifically didn't want to suggest a solution as I'm not expert enough to know what the best solution might be.
If, as Paul implies, it should be possible to run the standard library from a zipfile then it suggests that this should actually be a bug report.
brettcannon commentedon Jan 4, 2016
What Phil is after is a hook at the C level that will allow him to modify the sys module to add something to sys.meta_path to remove PathEntryFinder and put in his own importer (assuming he isn't simply freezing everything).
And yes, you should be able to run the stdlib from a zipfile, but the issue here is all the stuff Python imports as part of startup before the first bit of user Python code is executed, e.g. the encodings module. Since zipimport is added by importlib it gets special treatment to be usable ASAP. Since Phil wants to completely skew standard import mechanisms this does require a C hook.
What this all means is that Phil really needs https://www.python.org/dev/peps/pep-0432/ to happen so that he can easily get his hook point added into the startup sequence.
philthompson10 commentedon Jan 4, 2016
At the moment my importer does the same as zipimport and gets added to sys.path_hooks rather than sys.meta_path.
While waiting for the PEP, how about a table of (externally modifiable) importer installers that is worked through where _PyImportZip_Init() is currently called? By default _PyImportZip_Init() will be the only entry.
brettcannon commentedon Jan 4, 2016
I personally would rather wait until PEP-432 happens before adding any such hook. This sort of thing is extremely hard to test and the startup sequence for Python is already quite complicated and honestly too tightly bound to the magical sequence of steps it current has.
philthompson10 commentedon Jan 4, 2016
I don't see why it would be hard to test. The change suggested wouldn't alter the default behaviour at all.
brettcannon commentedon Jan 4, 2016
It might not change the default behaviour, but we still have to make sure it works and doesn't accidentally break. If we add official support for a C API to specify callbacks to execute during interpreter startup before any stdlib Python code then we are making a promise that it will work and so we have to worry about it even if the vast majority of users will never use it.
philthompson10 commentedon Jan 4, 2016
Understood, but the only promise here is to call a function (with no arguments, and returning no result).
[-]Request for Support for Embedding the Standard Library in an Executable[/-][+]Support embedding the standard library in an executable[/+]brettcannon commentedon Sep 11, 2020
Update: PEP-432 was withdrawn, so this issue is now languishing and not blocked on something else.