Repository navigation
PYTHONOPTIMIZE ignored in 3.7.0 when using custom launcher #78428
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/vstinner'
closed_at = <Date 2018-09-20.12:32:35.054>
created_at = <Date 2018-07-27.12:07:29.981>
labels = ['interpreter-core', 'type-bug', '3.7']
title = 'PYTHONOPTIMIZE ignored in 3.7.0 when using custom launcher'
updated_at = <Date 2019-03-06.11:51:55.385>
user = 'https://github.com/ronaldoussoren'bugs.python.org fields:
activity = <Date 2019-03-06.11:51:55.385>
actor = 'vstinner'
assignee = 'vstinner'
closed = True
closed_date = <Date 2018-09-20.12:32:35.054>
closer = 'ncoghlan'
components = ['Interpreter Core']
creation = <Date 2018-07-27.12:07:29.981>
creator = 'ronaldoussoren'
dependencies = []
files = ['47716', '47717']
hgrepos = []
issue_num = 34247
keywords = ['patch', '3.7regression']
message_count = 27.0
messages = ['322487', '322519', '322520', '322521', '322522', '322523', '322524', '322529', '322532', '322533', '322550', '322562', '323068', '323070', '323135', '323144', '323145', '323156', '325172', '325281', '325283', '325288', '325289', '325862', '325874', '325877', '337294']
nosy_count = 7.0
nosy_names = ['brett.cannon', 'ronaldoussoren', 'ncoghlan', 'vstinner', 'ned.deily', 'eric.snow', 'miss-islington']
pr_nums = ['8521', '8659', '9223', '9275', '12191']
priority = None
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue34247'
versions = ['Python 3.7']
Activity
ronaldoussoren commentedon Jul 27, 2018
Attached is a very basic launcher for python scripts. This script works as expected in Python 3.6, but not in 3.7.0.
In particular, the launcher sets the environment variable PYTHONOPTIMIZE before calling Py_Initialize and that setting is ignored.
I haven't tried testing with the tip of the tree yet.
ncoghlan commentedon Jul 28, 2018
This feels like a duplicate of https://bugs.python.org/issue31845 (see d7ac061 )
Is this definitely with the 3.7.0 release, or is it potentially with one of the earlier alphas before that fix was applied?
ncoghlan commentedon Jul 28, 2018
Ah, wait - I see a relevant difference: the test case for bpo-31845 uses Py_Main, *not* Py_Initialize (since it's a command line test, not an embedding test).
That means it's currently possible for the sequence of operations in
cpython/Python/pylifecycle.c
Line 913 in 7cbde0e
ncoghlan commentedon Jul 28, 2018
Comparing to Python 3.6, we can see that the code to update the C global flags from the environment variables used to live directly in _PyInitializeEx_Private:
cpython/Python/pylifecycle.c
Line 305 in 9d85856
In Python 3.7 that section (https://github.com/python/cpython/blob/3.7/Python/pylifecycle.c#L913 ) currently calls _PyCoreConfig_Read instead, which *only* updates the config struct - it doesn't update the C level global variables.
By contrast, Py_Main eventually calls
cpython/Modules/main.c
Line 1394 in 7cbde0e
ncoghlan commentedon Jul 28, 2018
Next steps:
ncoghlan commentedon Jul 28, 2018
Ned, I think we should consider this a release blocker for 3.7.1
ncoghlan commentedon Jul 28, 2018
This seems closely related to the work Victor is already pursuing to resolve bpo-34170 for Python 3.8 (e.g. 56b29b6 ).
The bpo-34170 changes are more invasive than we'd like for a maintenance release, but it would be preferable to keep at least any new Python 3.7 test cases somewhat aligned with their Python 3.8 counterparts.
ronaldoussoren commentedon Jul 28, 2018
I'm at the EuroPython sprints and currently working on a testcase for this. Should be a nice way to get back into actively contributing to CPython :-)
ronaldoussoren commentedon Jul 28, 2018
FYI: I've filed bpo-34255 while working on this, test_embed assumes that you're building in the source directory (which I usually don't do).
ronaldoussoren commentedon Jul 28, 2018
Interesting... pylifecycle.c uses code in main.c, I hadn't expected that.
If I've read the code correctly Py_Initialize won't look at PYTHONOPTIMZE at all (and other environment variables that have a command-line equivalent). Those are read in main.c:cmdline_get_env_flags, which is only called (indirectly) from Py_Main and _Py_UnixMain.
I'm note sure what's the correct solution for this.
Reading PYTHONOPTIMZE (and related variables) could be done
in _PyCoreConfig_Read, but that would then need to grow a flag to ensure
that this won't replace values calculated earlier in the implementation
of Py_Main
Writing back the global variables is probably best done using a new
function (lets _PyCoreConfig_SetGlobals), mirroring
pymain_set_global_config but using a _PyCoreConfig config structure
instead of the command-line structure used by the latter function.
6 remaining items
vstinner commentedon Aug 5, 2018
On my PR, Nick Coghlan wrote:
"We may want to tweak the porting note in What's New, as I believe this may read more environment variables at init time than the interpreter did previously, so embedding applications that want more complete control may now need to set Py_IgnoreEnvironmentFlag when they could previously get by without it."
#8659 (review)
I'm not sure about this, since it's really a bug of Python 3.7.0 and it's now fixed.
I let you decide if the issue should be closed or not, I leave it open.
[-]PYTHONOPTIMZE ignored in 3.7.0 when using custom launcher[/-][+]PYTHONOPTIMIZE ignored in 3.7.0 when using custom launcher[/+]ncoghlan commentedon Aug 5, 2018
My comment wasn't about the 3.7.0 -> 3.7.1 fix, but rather about the fact that I suspect that 3.7.1 is now going to respect some environment variables in this case that 3.6.x ignored. I don't know for sure though, since we didn't have a test case for this until you wrote one.
So I was thinking we could include a porting note along the lines of:
===============
===============
That note then serves two purposes:
Py_IgnoreEnvironmentFlag = 0just plain doesn't work properly in 3.7.0 (without their needing to find this bug for themselves)Py_IgnoreEnvironmentFlag = 1if their embedded interpreter is now being affected by environmental settings that it ignored in previous versionsned-deily commentedon Sep 12, 2018
PR 9223 adds Nick's proposed wording to the 3.7 What's News.
ned-deily commentedon Sep 13, 2018
New changeset 66755cb by Ned Deily in branch 'master':
bpo-34247: add porting note to 3.7 What's New (GH-9223)
66755cb
vstinner commentedon Sep 13, 2018
I'm not aware of the Python 3.6 issue ("ill-defined"): which env vars were not properly handled?
miss-islington commentedon Sep 13, 2018
New changeset 3050564 by Miss Islington (bot) in branch '3.7':
bpo-34247: add porting note to 3.7 What's New (GH-9223)
3050564
ned-deily commentedon Sep 13, 2018
I've merged Nick's suggested porting note so I'm going to remove the "release blocker" status. Can we also close this now?
ncoghlan commentedon Sep 20, 2018
The "ill-defined" in Python 3.6 relates to the fact that we never actually defined or tested which environment variables were read by Py_Main and which ones were read by Py_Initialize, since the majority of our tests only covered Py_Main (by launching a full Python subprocess).
Thus the only way to find out which environment variables fell into which category would be to read the Python 3.6 source code.
That's technically still the case for Python 3.7, but Victor's done some excellent refactoring work so it's much clearer in the code which vars may be updated if Py_Initialize is run a second time.
vstinner commentedon Sep 20, 2018
Oh ok, I see. Thanks for the explanation.
Well, for me the most important part is not the code, but tests. test_embed currently tests the following environment variables:
As you can see, the test suite is not complete yet. But since the tests are there, it shouldn't be hard to extend the test suite.
test_embed has InitConfigTests in 3.7 and master branches. I'm not interested to backport these tests to 3.6. I modified Python initialization deeply in 3.7, and the status of the code in 3.6 is "undefined". I don't think that it's worth it to backport these tests to 2.7 or 3.6. I success to focus on the master branch where we want to finish the implementation of the PEP-432 which should provide a better *public* API for that.
ncoghlan commentedon Sep 20, 2018
Yep, exactly - things are much improved already, thanks primarily to your work, and it seems likely they'll be even further improved by the time 3.8.0 comes around :)
vstinner commentedon Mar 6, 2019
New changeset 25d13f3 by Victor Stinner in branch 'master':
bpo-36142: PYTHONMALLOC overrides PYTHONDEV (GH-12191)
25d13f3