Repository navigation
test_importlib: test_multiprocessing_pool_circular_import() fails with "Too many open files" error on os.pipe() #87758
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/gpshead'
closed_at = <Date 2021-10-22.04:49:25.641>
created_at = <Date 2021-03-22.10:33:02.909>
labels = ['OS-mac', 'type-bug', 'tests', '3.10']
title = 'test_importlib: test_multiprocessing_pool_circular_import() fails with "Too many open files" error on os.pipe()'
updated_at = <Date 2021-10-22.04:49:25.640>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2021-10-22.04:49:25.640>
actor = 'gregory.p.smith'
assignee = 'gregory.p.smith'
closed = True
closed_date = <Date 2021-10-22.04:49:25.641>
closer = 'gregory.p.smith'
components = ['macOS', 'Tests']
creation = <Date 2021-03-22.10:33:02.909>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 43592
keywords = ['patch']
message_count = 13.0
messages = ['389301', '389303', '389334', '392245', '400835', '400836', '400837', '400838', '401569', '404665', '404714', '404719', '404720']
nosy_count = 7.0
nosy_names = ['gregory.p.smith', 'ronaldoussoren', 'vstinner', 'ned.deily', 'lukasz.langa', 'lielfr', 'miss-islington']
pr_nums = ['29127', '29144', '29145']
priority = 'normal'
resolution = 'fixed'
stage = 'commit review'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue43592'
versions = ['Python 3.10']
Activity
vstinner commentedon Mar 22, 2021
x86-64 macOS 3.x:
https://buildbot.python.org/all/#/builders/366/builds/969
Build triggered by the commit 88d9983 which changes PyImport_Import().
The buildbot is running on macOS 10.15.7 (Darwin Kernel Version 19.6.0) with a limit of 256 file descriptors.
The latest successful build is build 968, whereas the the RLIMIT_NOFILE resource soft limit was also set to 256:
https://buildbot.python.org/all/#/builders/366/builds/968
test.pythoninfo:
FAIL: test_multiprocessing_pool_circular_import (test.test_importlib.test_threaded_import.ThreadedImportTests)
----------------------------------------------------------------------
stdout:
---
---
stderr:
---
vstinner commentedon Mar 22, 2021
test_importlib only fails *randomly* on x86-64 macOS 3.x.
For example, the build 970 is a success (but orange): "Tests result: FAILURE then SUCCESS". test_importlib failed with too many open files, and then passed.
ned-deily commentedon Mar 22, 2021
FWIW, I don't recall ever seeing a "too many open files" failure for test_importlib on any of my macOS systems.
lielfr commentedon Apr 28, 2021
I have the same error when running the tests on my machine (Same macOS version).
vstinner commentedon Sep 1, 2021
x86-64 macOS 3.10:
https://buildbot.python.org/all/#/builders/681/builds/345
======================================================================
FAIL: test_multiprocessing_pool_circular_import (test.test_importlib.test_threaded_import.ThreadedImportTests)
----------------------------------------------------------------------
stdout:
---
---
stderr:
---
vstinner commentedon Sep 1, 2021
On this machine, RLIMIT_NOFILE soft limit is 256.
test.pythoninfo:
resource.RLIMIT_NOFILE: (256, 9223372036854775807)
platform.platform: macOS-11.5.2-x86_64-i386-64bit
socket.hostname: mattb-mbp2
All limits:
resource.RLIMIT_AS: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_CORE: (0, 9223372036854775807)
resource.RLIMIT_CPU: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_DATA: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_FSIZE: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_MEMLOCK: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_NOFILE: (256, 9223372036854775807)
resource.RLIMIT_NPROC: (2784, 4176)
resource.RLIMIT_RSS: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_STACK: (67104768, 67104768)
resource.pagesize: 4096
ambv commentedon Sep 1, 2021
Try setting the following:
$ sudo sysctl -w kern.maxfiles=65536 $ sudo sysctl -w kern.maxfilesperproc=65536 $ sudo ulimit -n 65536ambv commentedon Sep 1, 2021
Before you do it, check the defaults with:
I'm curious.
vstinner commentedon Sep 10, 2021
FYI the test is still failing with the same error on the billenstein-macos builder worker, x86-64 macOS 3.10:
https://buildbot.python.org/all/#/builders/681/builds/370
"OSError: [Errno 24] Too many open files"
gpshead commentedon Oct 21, 2021
macOS often has an unreasonably low default max number of file descriptors. This is easy to raise from user code, it's a soft limit.
#29127
is out to try that from regrtest itself.
gpshead commentedon Oct 22, 2021
New changeset 843b890 by Gregory P. Smith in branch 'main':
bpo-43592: Raise RLIMIT_NOFILE in test.libregrtest (GH-29127)
843b890
miss-islington commentedon Oct 22, 2021
New changeset 8f6aa48 by Miss Islington (bot) in branch '3.10':
bpo-43592: Raise RLIMIT_NOFILE in test.libregrtest (GH-29127)
8f6aa48
miss-islington commentedon Oct 22, 2021
New changeset 216c040 by Miss Islington (bot) in branch '3.9':
bpo-43592: Raise RLIMIT_NOFILE in test.libregrtest (GH-29127)
216c040