Repository navigation
Python sub-interpreter test #55123
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/vstinner'
closed_at = <Date 2011-04-30.12:21:19.771>
created_at = <Date 2011-01-15.13:38:37.736>
labels = ['interpreter-core', 'tests']
title = 'Python sub-interpreter test'
updated_at = <Date 2012-11-28.06:45:09.320>
user = 'https://github.com/pitrou'bugs.python.org fields:
activity = <Date 2012-11-28.06:45:09.320>
actor = 'eric.snow'
assignee = 'vstinner'
closed = True
closed_date = <Date 2011-04-30.12:21:19.771>
closer = 'vstinner'
components = ['Interpreter Core', 'Tests']
creation = <Date 2011-01-15.13:38:37.736>
creator = 'pitrou'
dependencies = []
files = ['20414', '21282', '21765', '21778']
hgrepos = []
issue_num = 10914
keywords = ['patch']
message_count = 33.0
messages = ['126328', '126380', '126382', '126384', '126385', '126401', '126403', '131295', '131296', '131302', '131303', '131321', '131360', '132106', '134334', '134335', '134336', '134404', '134405', '134413', '134417', '134421', '134423', '134425', '134510', '134515', '134606', '134855', '134857', '143464', '143498', '143518', '143519']
nosy_count = 12.0
nosy_names = ['loewis', 'amaury.forgeotdarc', 'ncoghlan', 'pitrou', 'vstinner', 'christian.heimes', 'benjamin.peterson', 'tarek', 'eric.araujo', 'grahamd', 'python-dev', 'eric.snow']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue10914'
versions = ['Python 3.2', 'Python 3.3']
Activity
pitrou commentedon Jan 15, 2011
Here is a prototype test for embedding and sub-interpreters.
amauryfa commentedon Jan 16, 2011
+1 for this kind of tests.
But I would not have their source in the "official" Modules directory. I expect that there will be several tests of this kind, each one with different modules to import, functions to run, global settings to change.
IMO the C code should be somewhere in the test directory; later we could use templates to generate source code for similar tests.
Is it possible to use distutils to compile this test, and remove it from the Makefile?
benjaminp commentedon Jan 16, 2011
There's already precedent for test modules in Modules/ what with xxmodule.c and _testcapimodule.c.
pitrou commentedon Jan 16, 2011
I'm not aware that distutils is able to compile applications rather than
extension modules, but that would certainly be better than a rule in the
Makefile, yes.
amauryfa commentedon Jan 16, 2011
In distutils.command.config, config().try_run(body) is probably what we need. Now, I don't know how to use it...
ncoghlan commentedon Jan 17, 2011
I like the idea of adding these tests as well.
Probably worth bringing up on python-dev - folks like Tarek should definitely be able to help with alternative ways of building a test application for this.
There is also always the possibility of writing it in Python and using ctypes to get at the necessary C API calls to create subinterpreters.
pitrou commentedon Jan 17, 2011
Well, I don't think calling Py_Initialize, Py_Finalize and Py_NewInterpreter from pure Python code is a very good idea.
Tarek, can you help? We'd like to use distutils to compile an executable embedding the interpreter.
python-dev commentedon Mar 17, 2011
New changeset a791dd7d51f3 by Antoine Pitrou in branch '3.2':
Issue bpo-10914: fix bogus memory management in Modules/getpath.c, leading to a possible crash when calling Py_SetPath()
http://hg.python.org/cpython/rev/a791dd7d51f3
pitrou commentedon Mar 17, 2011
Ouch, sorry, I got my commit message wrong.
merwok commentedon Mar 17, 2011
distutils sure knows how to build .o or .so files, but I don’t know about standalone executables (because I don’t know how the .o end up making an executable). If you want to try to do it, I would advise you not to use the config command but rather a compiler object directly. distutils.ccompiler.new_compiler() should give you an instance of a subclass of distutils.ccompiler.CCompiler suitable for your system, and then you can use help or the source to find what methods to call.
pitrou commentedon Mar 17, 2011
Well, config._link() seems to do what is needed here. If you think it is
bad to rely on it, perhaps we should "inline" its code somehow in the
test module.
merwok commentedon Mar 18, 2011
pitrou commentedon Mar 18, 2011
Ok, this is what I came up with to build an exe using distutils. At this point, the complication is downright silly and it doesn't even work under Windows, so we may prefer to go down the Makefile route instead.
10 remaining items
vstinner commentedon Apr 25, 2011
The problem is a little bit more complex. interp->codecs_initialized is set to 1 by _PyCodecRegistry_Init(), but the filesystem codec is not loaded yet. We need another variable: interp->fscodec_initialized.
fscodec_subinterpreter.patch fixes this issue.
Note: the patch replaces also PyErr_Print() by PyErr_PrintEx(0) to avoid a crash on error, if PyErr_PrintEx(0) is called before the sys module is initialized. For example, if PyModule_GetDict(_PyImport_FindBuiltin("builtins")) is NULL. Anyway, it's useless to set sys attributes on error, because sys is destroyed just after printing the error.
python-dev commentedon Apr 26, 2011
New changeset c8338cfa3578 by Victor Stinner in branch 'default':
Issue bpo-10914: Py_NewInterpreter() uses PyErr_PrintEx(0)
http://hg.python.org/cpython/rev/c8338cfa3578
New changeset d3af2a2b621b by Victor Stinner in branch 'default':
Issue bpo-10914: Initialize correctly the filesystem codec when creating a new
http://hg.python.org/cpython/rev/d3af2a2b621b
vstinner commentedon Apr 26, 2011
test_capi pass on x86 debian parallel 3.x: I close this issue again :-)
pitrou commentedon Apr 27, 2011
Victor, the fix needs to go into 3.2 as well.
python-dev commentedon Apr 30, 2011
New changeset 5a983773c09a by Victor Stinner in branch '3.2':
Issue bpo-10914: Py_NewInterpreter() uses PyErr_PrintEx(0)
http://hg.python.org/cpython/rev/5a983773c09a
New changeset 2caf82aee7a4 by Victor Stinner in branch '3.2':
Issue bpo-10914: Initialize correctly the filesystem codec when creating a new
http://hg.python.org/cpython/rev/2caf82aee7a4
vstinner commentedon Apr 30, 2011
Oh, I thought that Modules/_testembed.c was only in 3.3. Anyway, it is a real bug in 3.2, and it is now fixed (I backported the 2 fixes in 3.2).
merwok commentedon Sep 3, 2011
BTW, about this comment in the code:
# XXX only tested under Unix checkouts
It is possible to install the _testembed.c file alongside xxmodule.c and compile it at test time. To test under Windows, I’ve re-read your first patch using distutils and it’s not that long or hard to read after all.
pitrou commentedon Sep 4, 2011
We probably could, using the approach from the distutils patch. But it feels really quirky to me. Granted, the Makefile is quirky as well.
Except that it didn't work under Windows...
merwok commentedon Sep 5, 2011
pitrou commentedon Sep 5, 2011
Agreed, but it's not something that I know how to do :)
(and yet, of course, it would be good to test that embedding works under
Windows too, as well as under Unix installs)