_PyImport_SwapPackageContext would be helpful if exposed #139227
Description
Activity
This was discussed in https://discuss.python.org/t/43396 for another loader, but we found a workaround for that use case.
IMO it would be helpful to discuss Nuitka's case too.I think Meta's case only involved statically linked extensions, but this applies to all extensions inside or with submodules in Nuitka's case. I do agree that we should get
_PyImport_SwapPackageContextasPyUnstable_Import_SwapPackageContexteventually, but that ship has already sailed for 3.14.Is it Nuitka-compiled modules only (which could be fixable by using multi-phase-init), or also third-party ones?
Third-party ones.
OK, that's not as easy. A 3.14 change is in the hands of the RM no, I guess.
Other than that, it looks like the process is broken. How did this behaviour manage to survive the beta period and three RCs?
Other than that, it looks like the process is broken. How did this behaviour manage to survive the beta period and three RCs?
This has been the case since 3.12, as far as I can tell. Again, Nuitka has been working around it using some hacks, but we've been hesitant to ask for upstream changes as we're relying on internals. It'd be fine if it didn't make it to 3.14, but just a little inconvenient.
It'd be fine if it didn't make it to 3.14, but just a little inconvenient.
If that's the case, you can pretty much close the PR -- a code change after the final RC should need an exceptionally good reason.
Instead, shall we aim for Unstable API? That means it needs documentation & tests.
The new docs single-phase init should make it easier to explain why you'd need this function.I can just update the PR to use the unstable API.
Instead, shall we aim for Unstable API? That means it needs documentation & tests.
Yeah, but that makes it a lot more difficult to justify, and adds more maintenance burden on us (CPython, not Nuitka -- I've been back and forth with "we"/"us"). Explicit tests for something like this are going to be complex and won't catch much anyway, because if we were to break
_PyImport_SwapPackageContextthen our import system would fail anyway.I'm not opposed to making it
PyUnstable, but we should perhaps be more relaxed with our requirements for unstable APIs here.Would GH-139196 help here?
If we want 3rd parties like Nuitka to use something, we should expose it as a proper interface, document how it should be used, and test it so that we don't break it in a patch release.
if we were to break _PyImport_SwapPackageContext then our import system would fail anyway.
Not necessarily. We could, for example, add an argument to it and update all the callers.
This is more urgent now that 3.15 breaks the ability to access
pkgcontextfrom outside CPython.Would #139196 help here?
It could, but the devil's in the details. Nuitka does a bunch of PGO shenanigans and adds other shims that would probably break if we switched to that. At this point, it'd be much simpler if we simply had access to
_PyImport_SwapPackageContext.That said, I don't think I can reasonably come up with a test case to exercise a
PyUnstable_Import_SwapPackageContext, at least short of writing an entire importer, which is not ideal for a single unstable C API function. Maybe that's indicative of this being too low-level for the unstable API? I'm not sure.I don't understand well the purpose of this "package context". It's made of multiple parts:
_PyRuntime.imports.pkgcontextstatic _Py_thread_local const char *pkgcontext = NULL;inPython/import.cconst char* _PyImport_ResolveNameWithPackageContext(const char *name)const char* _PyImport_SwapPackageContext(const char *newcontext)
To understand how it works, I commented out
_PyImport_SwapPackageContext()calls inPython/importdl.c. Problem: the whole test suite pass without it :-( It doesn't help me to understand its purpose... So this API is not tested by Python test suite.From what I understood, the "package context" is used to override the module name in
_PyModule_CreateInitialized(). For example,PyModuleDef.m_nameis"module", but if the package context is set to"package.module", the module name becomes"package.module".Why
PyModuleDef.m_nameis not"package.module"at the beginning?Note:
_PyRuntime.imports.pkgcontextis no longer used and can be removed. I wrote PR gh-151490 to remove it._PyImport_SwapPackageContext()stores a raw pointer to a bytes string without copying the string. If we expose it, we should clearly describe this limitation: the caller must ensure that the pointer remains valid until the next_PyImport_SwapPackageContext()call.
_PyImport_RunModInitFunc()calls_PyImport_SwapPackageContext()withinfo->newcontextargument:/* Package context is needed for single-phase init */ const char *oldcontext = _PyImport_SwapPackageContext(info->newcontext); PyObject *m = p0(); _PyImport_SwapPackageContext(oldcontext);
The info is a
struct _Py_ext_module_loader_info*structure which is set in a caller function:int _Py_ext_module_loader_info_init(struct _Py_ext_module_loader_info *p_info, PyObject *name, PyObject *filename, _Py_ext_module_origin origin) { ... info.name = Py_NewRef(name); info.name_encoded = get_encoded_name(info.name, &info.hook_prefixes); if (info.name_encoded == NULL) { ... return -1; }
_PyImport_SwapPackageContext()just stores a raw pointer to a string, it doesn't copy the string:const char * _PyImport_SwapPackageContext(const char *newcontext) { const char *oldcontext = PKGCONTEXT; PKGCONTEXT = newcontext; return oldcontext; }
So
PKGCONTEXTremains valid untilinfo->nameis valid (strong reference).I was able to implement this in Nuitka using 3.15's new
PyImport_CreateModuleFromInitfuncAPI, so I no longer need this. Thanks for the insight, both of you!Reacted by Victor Stinner
Feature or enhancement
In Nuitka, we have our own meta-path-based loader, which loads packages that contain extension modules with custom code that emulates the load process. For an extension module in a package, we used to set
_Py_PackageContext, which has since been removed. Currently,_PyRuntime.imports.pkgcontextis not used because we fall back to a thread-localpkgcontextvariable, which Nuitka cannot access on Windows or where static linking to Python is not available.This leads to some issues when Nuitka goes to load extensions that are stored in a submodule or create submodules, because the package context is not correct. For example, an extension module under
x.ywill create submodulezasy.zrather thanx.y.z. There are currently some workarounds to repair the damage, but this is tedious and sometimes unreliable. If_PyImport_SwapPackageContextwere exposed, Nuitka could just call that and makePyModule_Createknow what package it lives in.Nuitka doesn't need
_PyImport_SwapPackageContextto be exposed as public orPyUnstable, we would just like to have it available in the ABI underPyAPI_FUNC. It would be very helpful if this could happen before 3.14.0, but that's up to Hugo.Linked PRs
_PyImport_SwapPackageContextin the ABI #139228