Allow for built-in modules to be submodules

Bug report

If you statically link a namespaced extension (such as numpy.core._multiarray_umath) and add it to the inittab, it will not get loaded properly. In this case, the path to the extension is passed to the BultinImporter.find_spec method which causes it to return None. The path is a pointer to the namespace of the extension.

One possible fix would be to check for a . in the name of the extension being loaded. If it contains that, consider it to be a fully-qualified import path and check to see if that fully-qualified name is in the builtins list before moving on to other checks.

Your environment

CPython 3.10 (although I believe it's the same in newer versions)
Linux x86 and Wasm

Linked PRs

@brettcannon

In GH-102768 @brettcannon wrote:

As discovered while trying to get NumPy statically compiled into CPython for WASI, the built-in importer automatically rejects any built-in module name with a .. But honestly I don't know if there's a need to be so strict since there technically isn't a reason why a built-in module couldn't be a submodule.

@kesmit13

@kesmit13 thanks for opening the issue.

numpy and numpy.core are normal packages, with only numpy.core._multiarray_umath being a built-in module, right?

the built-in importer automatically rejects any built-in module name with a .

The root issue here is that path is set, which happens because the parent module is a package.

cpython/Lib/importlib/_bootstrap.py

Lines 1262 to 1269 in 405739f

    
               try: 
        
                   path = parent_module.__path__ 
        
               except AttributeError: 
        
                   msg = f'{_ERR_MSG_PREFIX} {name!r}; {parent!r} is not a package' 
        
                   raise ModuleNotFoundError(msg, name=name) from None 
        
               parent_spec = parent_module.__spec__ 
        
               child = name.rpartition('.')[2] 
        
           spec = _find_spec(name, path)

I think the question we want to ask here is if we want to support having a built-in module be a submodule of a normal package? If so, we should simply remove the path check from BultinImporter.find_spec.

Your comments about numpy.core._multiarray_umath are correct. The issue that I'm running into is trying to create a statically linked Python binary that includes numpy and pandas. In order to do this, I have to use PyImport_AppendInittab with the C extensions from those packages.

Thanks, I think it makes sense to drop this limitation. I don't think there's anything else preventing built-in modules from being submodules other than the path check, so removing that check should be all that's needed.

pythonGH-102700: allow built-in modules to be submodules

GH-102700: allow built-in modules to be submodules (GH-103162)

pythonGH-102700: allow built-in modules to be submodules (pythonGH-10…

[3.11] GH-102700: allow built-in modules to be submodules (GH-103162) (…

pythonGH-102700: allow built-in modules to be submodules (pythonGH-10…

Navigation Menu

Repository navigation

Allow for built-in modules to be submodules #102700

Bug report

Your environment

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	try:
	path = parent_module.__path__
	except AttributeError:
	msg = f'{_ERR_MSG_PREFIX} {name!r}; {parent!r} is not a package'
	raise ModuleNotFoundError(msg, name=name) from None
	parent_spec = parent_module.__spec__
	child = name.rpartition('.')[2]
	spec = _find_spec(name, path)

Allow for built-in modules to be submodules #102700

Description

Bug report

Your environment

Linked PRs

Activity

FFY00 commented on Mar 16, 2023

FFY00 commented on Mar 16, 2023

kesmit13 commented on Mar 17, 2023

FFY00 commented on Mar 17, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions