Repository navigation
C Extension import limit #82778
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2021-02-03.18:08:03.183>
created_at = <Date 2019-10-26.19:48:20.615>
labels = ['3.8', '3.7', 'library', '3.9', 'performance']
title = 'C Extension import limit'
updated_at = <Date 2021-02-03.18:08:03.182>
user = 'https://bugs.python.org/KevinSchlossser'bugs.python.org fields:
activity = <Date 2021-02-03.18:08:03.182>
actor = 'steve.dower'
assignee = 'none'
closed = True
closed_date = <Date 2021-02-03.18:08:03.183>
closer = 'steve.dower'
components = ['Distutils']
creation = <Date 2019-10-26.19:48:20.615>
creator = 'Kevin Schlossser'
dependencies = []
files = ['48918']
hgrepos = []
issue_num = 38597
keywords = ['patch']
message_count = 23.0
messages = ['355425', '355578', '356824', '356887', '356892', '359314', '359395', '359403', '362529', '362549', '362557', '362653', '362756', '362793', '362795', '362796', '363067', '363080', '363157', '363222', '363225', '363226', '386268']
nosy_count = 13.0
nosy_names = ['paul.moore', 'scoder', 'tim.golden', 'eric.araujo', 'zach.ware', 'steve.dower', 'dstufft', 'p-ganssle', 'miss-islington', 'Kevin Schlossser', 'Yurii Leonov', 'xinfazhu', 'P12 Pfx']
pr_nums = ['18724', '18758', '18759']
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue38597'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']
Activity
KevinSchlossser commentedon Oct 26, 2019
System
Windows 7 x64 SP2
Ram 16GB
6 Core AMD @ 3.2ghz
CPython 3.7.2
C Extension (pyd) import cap.
There seems to be a cap on the number of extensions that a package is able to contain. I am able to import 123 extension modules that my package has but when i go to import number 124 i get the following traceback
ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed.
these extension modules are part of my package, importing an extension module from another package does not change this behavior. it is only when I import the 124th extension that is in my package does it occur.
When I change the order of the imports the error does not follow the import. I end up getting the same error when the 124th extension gets loaded doesn't matter what extension it is.
I have tried to see if maybe it was a module limit and I spread the imports across multiple files and it still fails when the 124th gets loaded. I also tried imp.load_dynamic and importlib.import_module and the same error occurs.
If there is a way to work around this limitation it would be very helpful.
zooba commentedon Oct 28, 2019
There is an implicit cap due to the C runtime (specifically vcruntime140.dll) allocating fibre local storage on load, which seems like the one you are hitting. However, we discovered this before the first 3.5 release and fixed it.
How are you compiling your packages? Or where are you getting them from?
It's possible that someone else is building and statically linking the C runtime, which will cause this, but distutils (and hence CPython) should not be doing it by default.
YuriiLeonov commentedon Nov 17, 2019
Looks like I have same problem for Windows 10 (version 1809, build - 17763.864).
I created repository with steps for reproducing - https://github.com/Yuriy-Leonov/cython_imports_limit_issue
zooba commentedon Nov 18, 2019
Could you share just one of your .pyd files?
Without being able to see whether they are compiled incorrectly, it's hard to be sure whether this is the same cause as before. It certainly looks like distutils is still going to link correctly.
14 remaining items
JoeXinfa commentedon Feb 27, 2020
FYI when I reduced my package to 106 extension modules, I could run without the DLL error.
zooba commentedon Feb 27, 2020
Thanks, it does seem like it's finding the correct MSVC, but is not finding the redistributable DLL:
The "/MT" means to statically the CRT, and then we use link settings later to dynamically link the part installed in the OS (see https://stevedower.id.au/blog/building-for-python-3-5-part-two for more details).
The problem seems to be that this glob in distutils can't find the path listed below it:
"2019\BuildTools\VC\redist\MSVC\**\x64\Microsoft.VC14*.CRT\vcruntime140.dll"
"2019\BuildTools\VC\Redist\MSVC\14.24.28127\x64\Microsoft.VC142.CRT\vcruntime140.dll"
We swallow a lot of errors while doing this glob, so I assume there's something about your install that's affecting it (maybe related to it being BuildTools rather than Community or higher?)
Could you try running this code and see what result/error you get:
zooba commentedon Feb 27, 2020
Hmm, just ran it on my own and it's finding the OneCore file first, which is not a problem (yet) but could become one.
So we at least need to replace the "**" with a "*" in Lib/distutils/_msvccompiler.py#L109. Not sure if that will help in this other case or not.
zooba commentedon Mar 1, 2020
In thinking about this, I think the best way forward is to just remove the logic that might statically link the initialization code, and instead commit to CPython releases always including vcruntime140.dll even if we switch to a newer version one day.
Hopefully third party distributions will do the same, though it should only matter for ABI3 modules that are not recompiled for the newer versions.
JoeXinfa commentedon Mar 1, 2020
Steve, don't know if you still need it but here is what you requested. Sorry for the slow move (I was working on something else). Seems mine is finding the x64 before the OneCore, though I don't know the significance.
Python 3.7.1 (default, Dec 10 2018, 22:54:23) [MSC v.1915 64 bit (AMD64)] on win32
In[2]: import glob
In[3]: glob.glob(r"C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\redist\MSVC\**\x64\Microsoft.VC14*.CRT\vcruntime140.dll", recursive=True)
Out[3]:
['C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\redist\\MSVC\\14.24.28127\\x64\\Microsoft.VC142.CRT\\vcruntime140.dll',
'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\redist\\MSVC\\14.24.28127\\onecore\\x64\\Microsoft.VC142.CRT\\vcruntime140.dll']
zooba commentedon Mar 2, 2020
Thanks. That basically confirms that something is interfering with distutils. Skipping the check entirely should avoid it.
zooba commentedon Mar 3, 2020
New changeset ce3a498 by Steve Dower in branch 'master':
bpo-38597: Never statically link extension initialization code on Windows (GH-18724)
ce3a498
miss-islington commentedon Mar 3, 2020
New changeset 8a5f7ad by Steve Dower in branch '3.7':
[3.7] bpo-38597: Never statically link extension initialization code on Windows (GH-18724) (GH-18759)
8a5f7ad
miss-islington commentedon Mar 3, 2020
New changeset 0d20364 by Miss Islington (bot) in branch '3.8':
bpo-38597: Never statically link extension initialization code on Windows (GH-18724)
0d20364
zooba commentedon Feb 3, 2021
Distutils is now deprecated (see PEP-632) and all tagged issues are being closed. From now until removal, only release blocking issues will be considered for distutils.
If this issue does not relate to distutils, please remove the component and reopen it. If you believe it still requires a fix, most likely the issue should be re-reported at https://github.com/pypa/setuptools