Skip to content

subprocess: Prefer close_range() to procfs-based fd closing #92301

Closed

Description

@izbyshev

subprocess gained the ability to use close_range() in #84603, however, it's only used as a fallback if procfs is not available, which means that at least on Linux it's almost never used. Its performance benefits are significant (see numbers at the end). I propose to use it by default, with procfs being the second option.

This requires some extra changes because _Py_closerange() was designed to be standalone and tries all available methods, including brute force, and also because it's not required to be async-signal-safe, but we must preserve async-signal-safety of fd closing code on Linux. I'll open a PR with a proposed implementation.

Performance comparison that I did on a test version with the following script (may require you to raise the fd limit in your shell):

import os, subprocess, sys, timeit
from resource import *

soft, hard = getrlimit(RLIMIT_NOFILE)
setrlimit(RLIMIT_NOFILE, (hard, hard))

num_fds, num_iter = map(int, sys.argv[1:3])

for i in range(num_fds):
    os.open('/dev/null', os.O_RDONLY)

print(timeit.timeit(lambda: subprocess.run('/bin/true'), number=num_iter))

Before:

./python test.py 100 100
0.11497399304062128
./python test.py 1000 100
0.3999998280778527
./python test.py 10000 100
3.19203062588349

After:

./python test.py 100 100
0.07936713611707091
./python test.py 1000 100
0.09550889488309622
./python test.py 10000 100
0.288825839292258

Activity

  1. added a commit that references this issue on May 5, 2022
  2. gpshead commented on May 5, 2022

    @gpshead
    Member

    Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    performancePerformance or resource usagetype-featureA feature request or enhancement

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions