No way to recover original argv with python -m #58416
Description
| BPO | 14208 |
|---|---|
| Nosy | @warsaw, @brettcannon, @birkenfeld, @rhettinger, @ncoghlan, @pitrou, @jwilk, @merwok, @florentx, @bdarnell |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2017-03-20.06:34:44.534>
created_at = <Date 2012-03-06.06:53:22.908>
labels = ['type-feature']
title = 'No way to recover original argv with python -m'
updated_at = <Date 2017-03-20.06:34:44.505>
user = 'https://github.com/bdarnell'bugs.python.org fields:
activity = <Date 2017-03-20.06:34:44.505>
actor = 'ncoghlan'
assignee = 'none'
closed = True
closed_date = <Date 2017-03-20.06:34:44.534>
closer = 'ncoghlan'
components = []
creation = <Date 2012-03-06.06:53:22.908>
creator = 'Ben.Darnell'
dependencies = []
files = []
hgrepos = []
issue_num = 14208
keywords = []
message_count = 13.0
messages = ['155002', '155003', '155044', '155053', '155073', '155102', '179845', '179852', '179855', '179856', '179882', '180455', '289874']
nosy_count = 12.0
nosy_names = ['barry', 'brett.cannon', 'georg.brandl', 'rhettinger', 'ncoghlan', 'pitrou', 'jwilk', 'eric.araujo', 'Arfrever', 'flox', 'Ben.Darnell', 'danielsh']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue14208'
versions = ['Python 3.4']
Activity
bdarnell commentedon Mar 6, 2012
I have a script which attempts to re-invoke itself using sys.argv, but it fails when run with "python -m package.module". The problem is that the handling of -m (via the runpy module) rewrites sys.argv as if it were run as "python package/module.py", but the two command lines are not equivalent: With -m the current directory is inserted at the head of sys.path, but without -m it's the directory containing module.py. The net effect is that the initial run of "python -m package.module" works as expected, but when it re-runs itself as "python package/module.py" the imports from module.py are effectively relative instead of absolute.
One possible solution would be to provide an immutable sys.__argv__ (by analogy with sys.__stdout__ and friends).
birkenfeld commentedon Mar 6, 2012
I agree this would be useful. It would be even more useful to have an __argv__ that includes all command-line flags given to Python, such as -Wi.
ncoghlan commentedon Mar 7, 2012
This is closely related to PEP-395, since multiprocessing currently hits the same issue in trying to figure out the correct setting for sys.argv0.
I quite like the sys.__argv__ idea for preserving the *exact* underlying command line (Alex Gaynor was recently asking for something similar to this).
In the meantime, it should be possible to work around the problem by running the affected subprocess invocations (i.e. when __main__.__package__ exists and is not empty) with something like:
rhettinger commentedon Mar 7, 2012
Instead of sys.__argv__, may I suggest sys.argv_original or somesuch.
ncoghlan commentedon Mar 7, 2012
In framing a question for Raymond regarding his preference for avoiding the __argv__ name, I realised I agreed with him. My reasoning is that, when a Python process starts, sys.stdin is sys.__stdin__, sys.stdout is sys.__stdout__ and sys.stderr is sys.__stderr__. The dunder versions capture the original values as created by the interpreter initialisation, not the raw OS level file descriptors.
The new attribute proposed here is different - it's not an immutable copy of the original value of sys.argv, it's a *different* sequence altogether. The analogy with the standard stream initial value capture created by the use of sys.__argv__ would actually be misleading rather than helpful.
For the same reason, Raymond's specific argv_original suggestion doesn't really work for me. Alas, I can think of several other possible colours for that particular bikeshed (such as argv_os, argv_main, argv_raw, argv_platform, argv_executable) without having any particular good way of choosing between them.
birkenfeld commentedon Mar 7, 2012
I agree. Maybe I may throw "full_argv" or "executable_argv" (i.e. to be used with exec([sys.executable] + sys.executable_arg)) in the air?
ncoghlan commentedon Jan 13, 2013
For PEP-432, I'm proposing to expose this as sys._configuration.raw_argv
pitrou commentedon Jan 13, 2013
I think sys.raw_argv would be as good. It's not really (not only) a
configuration item.
ncoghlan commentedon Jan 13, 2013
I'm not seeing a good justification for doing anything more, though:
Elevating it to part of the language standard doesn't make sense, while exposing it as an implementation feature of the CPython initialisation process seems more reasonable.
pitrou commentedon Jan 13, 2013
I expect it to be useful in the "launch (almost) the same command in the
same way" situation.
Not that it happens often :-)
danielsh commentedon Jan 13, 2013
Antoine Pitrou wrote on Sun, Jan 13, 2013 at 10:19:20 +0000:
What about the "launch a different command with the same interpreter
flags" use-case? For example, having 'python -E foo.py --foooptions'
want to execute 'python -E bar.py'.
Perhaps it'll be cleaner to expose in state_argv ['python', '-E']; in
sys.argv ['foo.py', '--foooptions']; and have scripts that want to exec
themselves use an idiomatic os.execv(sys.executable, sys.state_argv + sys.argv).
tjguk commentedon Jan 23, 2013
My use case is the reloader or restarter. I've initially fallen foul of this when using the cherrypy reloader (which does an execv by building from sys.executable + sys.argv) but I also have web services running which I'd like to restart remotely by forcing them to execv themselves.
It's trivial to retrieve the original command line from the Windows API, shlex.split it and pass it along to execv so this is hardly a showstopper. But it would be useful to have something equivalent built-in.
ncoghlan commentedon Mar 20, 2017
A few updates here:
For the specific case of
python -m, the original argument has been available as__main__.__spec__.namesince Python 3.4Also since Python 3.4, the
multiprocessingmodule has correctly handled the -m switch. For more details, see https://docs.python.org/3/whatsnew/3.4.html#multiprocessing and the linked issues.However, there are still some cases where it is potentially useful to have access to the full details of how the host Python runtime was invoked, rather than just the arguments that were left after CPython's runtime processing was completed. I filed bpo-29857 as a new RFE specifically suggesting a
sys._raw_argvattribute addressing that question.