os.cpu_count() should return a count assigned to a container or that the process is restricted to

@vstinner

BPO	36054
Nosy	@vstinner, @giampaolo, @tiran, @jab, @methane, @matrixise, @corona10, @Zheaoli, @mcnelsonphd, @NargiT, @Redeyed

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-02-20.16:56:22.384>
labels = ['interpreter-core', '3.8', '3.7', 'library', 'performance']
title = 'On Linux, os.count() should read cgroup cpu.shares and cpu.cfs (CPU count inside docker container)'
updated_at = <Date 2021-10-11.14:25:34.762>
user = 'https://github.com/keirlawson'

bugs.python.org fields:

activity = <Date 2021-10-11.14:25:34.762>
actor = 'corona10'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core', 'Library (Lib)']
creation = <Date 2019-02-20.16:56:22.384>
creator = 'keirlawson'
dependencies = []
files = []
hgrepos = []
issue_num = 36054
keywords = []
message_count = 24.0
messages = ['336117', '336126', '336146', '336148', '336149', '338724', '338778', '338780', '338783', '339401', '339404', '339429', '339439', '353690', '364894', '364898', '364901', '364902', '364916', '365071', '365075', '366811', '403622', '403654']
nosy_count = 12.0
nosy_names = ['vstinner', 'giampaolo.rodola', 'christian.heimes', 'jab', 'methane', 'matrixise', 'corona10', 'Manjusaka', 'mcnelsonphd', 'galen', 'nargit', 'RedEyed']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue36054'
versions = ['Python 3.7', 'Python 3.8']

There appears to be no way to detect the number of CPUs allotted to a Python program within a docker container. With the following script:

import os

print("os.cpu_count(): " + str(os.cpu_count()))
print("len(os.sched_getaffinity(0)): " + str(len(os.sched_getaffinity(0))))

when run in a container (from an Ubuntu 18.04 host) I get:

docker run -v "$PWD":/src/ -w /src/ --cpus=1 python:3.7 python detect_cpus.py
os.cpu_count(): 4
len(os.sched_getaffinity(0)): 4

Recent vesions of Java are able to correctly detect the CPU allocation:

docker run -it --cpus 1 openjdk:10-jdk
Feb 20, 2019 4:20:29 PM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
| Welcome to JShell -- Version 10.0.2
| For an introduction type: /help intro

jshell> Runtime.getRuntime().availableProcessors()
$1 ==> 1

I would like to work on this issue.

so, I have also tested with the last docker image of golang.

docker run --rm --cpus 1 -it golang /bin/bash

here is the code of golang:

package main

import "fmt"
import "runtime"

func main() {
cores := runtime.NumCPU()
fmt.Printf("This machine has %d CPU cores.\n", cores)
}

Here is the output

./demo
This machine has 4 CPU cores.

When I try with grep on /proc/cpuinfo
I get this result

grep processor /proc/cpuinfo -c
4

I will test with openjdk because it's related to Java and see if I can get the result of 1

ok, I didn't see your test with openjdk:10, sorry

I believe this is related to this ticket: https://bugs.python.org/issue26692

Looking at Java's implementation it seems like they are checking if cgroups are enabled via /proc/self/cgroup and then if it is parsing the cgroup information out of the filesystem.

I am really sorry but I thought to work on this issue but it's not the case. Feel free to submit a PR.

I think that I may work on a PR for this issue. Is there anybody has worked on it ？

Hi Manjusaka,

Could you explain your solution, because I have read the code of
openjdk, (C++) and I am going to be honnest, it was not complex but not
really clear.

Also, if you need my help for the review or for the construction of your
PR, I can help you.

Have a nice day,

Stéphane

Hi Stéphane

Thanks a lot!

In my opinion, I would like to make an independent library that name is cgroups. For ease of use and compatibility, I think it's better than combining code with the os module.

Thanks for you working!

Manjusaka

Hi Stéphane:

I have checked the JVM implantation about container improvements. I confirm that maybe we need a new Libary for container environment. I don't think that combine it into the os module is a good idea. I will make a PR during this week.

What matters is how many possible cores could execute code for this process in parallel.

Is there a reliable way to retrieve this information?

Yes, strictly speaking cgroups shares/weight has nothing to do with parallelism or number of CPUs. This cgroups mechanism is a way of specifying a proportion of the kernel scheduler's time slice that a cgroup can consume. A lot of people mistakenly believe that by specifying cpus=N to their container runtime they are asking for N CPUs. What they are actually asking for is N CPUs worth of time from the scheduler. They are free to consume those resources using as few or many threads as they want (subject to additional cgroups limitations) and the kernel will schedule work on as few or as many physical CPU cores as it chooses.

However, it is extremely popular to size thread/process to match the proportional CPU count given to the container runtime. And container runtimes make it easy to just divide the cgroups share by 1024 to deduce that value. So everyone does that. It mostly just works for the purposes people use it for.

Now if container runtimes exposed the resource settings to containers (by injecting an environment variable perhaps?), that is something I may potentially use. But at the same time, cgroups feels more canonical. We also have to consider that dynamic resource sizing for containers/cgroups is a thing after all! I can't imagine the maintainers of container runtimes will want to try to propagate mutable state into the container. It is far easier for them to do nothing and just tell end-users to query cgroups.

If cpu_count() starts parsing cgroups, I would suggest using the minimum between what's announced in cgroups and what's seen by taking in account CPU affinity: that's what does pylint.

See #80235 (comment) and the pointed code:

return min(cpu_share, cpu_count)

where cpu_count is len(sched_getaffinity(0)).

@indygreg

What I dislike here is that the ratio converting cgroups CPU shares to a number of logical CPUs is not standardized.

@indygreg proposes to announce this constant on a command line option / env var to solve this issue: #109595 (comment).

Maybe 1024 can be used by default, but it's important that it's possible to configure this value.

FYI, now we provide a way to restrict CPU resources from #109595.
This can be the alternative way to handle this issue.

After reading a lot about Linux cgroups CPU share, I don't think that they should be used in os.cpu_count() or os.process_cpu_count(). I close the issue.

The relationship between a CPU share and a number of CPU is not clear and caused a lot of headaches to many people.

Obviously, some people have different opinion and I suggest them to implement something similar to pylint or create a project on PyPI, get feedback from other users, and then come back with a strong rationale to convince why it's very important to add it to Python :-)

Maybe later we can consider to add a "get_cgroups_cpu_share()" function, but I'm not convinced about using it to compute cpu_count() or process_cpu_count().

@vstinner

@vstinner os not possible to define os.cpu_count() based on CPU share. IMHO should only care about quota(CPU limit), because is the only information that can be easily used to define the os.cpu_count().

This project can be useful https://github.com/uber-go/automaxprocs

IMHO should only care about quota(CPU limit)

I mean that in general, Linux cgroups CPU limits are not directly related to a whole number of CPU and so it's hard to use such limit to update os.cpu_count() or os.process_cpu_count().

Please create a PyPI project to parse Linux cgroups and see how it goes ;-)

So far, since 2019, I didn't see any clear implementation of "cpu_count" based on Linux cgroups. The closest thing that I saw so far was pylint code, but I don't think that we should do something like that in Python stdlib.

Is there a TL&DR for anyone coming to this thread from google?

I.e.; What is the canonical way to find the (lower) CPU core number due to cgroup/container resource limitations?

I.e.; What is the canonical way to find the (lower) CPU core number due to cgroup/container resource limitations?

It was decided to not handle this problem in Python. You should use a 3rd party project for that.

Can we provide some example, follow up links that could suggest the one or other way to solve this in our application code?

Or can the environment solve this with an approach similar to #109595 via an env variable? Will ask in that thread over there 👍

Or can the environment solve this with an approach similar to #109595 via an env variable? Will ask in that thread over there 👍

You can use PYTHON_CPU_COUNT env var to override os.cpu_count(): https://docs.python.org/dev/using/cmdline.html#envvar-PYTHON_CPU_COUNT

Can we provide some example, follow up links that could suggest the one or other way to solve this in our application code?

I don't think that we should provide examples in Python documentation. This problem is not solved, there are only heuristics, it depends on your environment and other stuff.

For what it's worth, in a Slurm job context you can use os.environ["SLURM_CPUS_ON_NODE"] or with SLURM_CPUS_PER_TASK to get a more accurate number of available cores.

https://slurm.schedmd.com/sbatch.html#OPT_SLURM_CPUS_ON_NODE

Navigation Menu

Repository navigation

os.cpu_count() should return a count assigned to a container or that the process is restricted to #80235

31 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

os.cpu_count() should return a count assigned to a container or that the process is restricted to #80235

Description

Activity

keirlawson commented on Feb 20, 2019

matrixise commented on Feb 20, 2019

matrixise commented on Feb 20, 2019

matrixise commented on Feb 20, 2019

keirlawson commented on Feb 20, 2019

matrixise commented on Mar 24, 2019

Zheaoli commented on Mar 25, 2019

matrixise commented on Mar 25, 2019

Zheaoli commented on Mar 25, 2019

Zheaoli commented on Apr 3, 2019

31 remaining items

vstinner commented on Sep 22, 2023

indygreg commented on Sep 22, 2023

vstinner commented on Sep 22, 2023

vstinner commented on Sep 22, 2023

corona10 commented on Oct 12, 2023

vstinner commented on Oct 12, 2023

robsonpeixoto commented on Oct 12, 2023

vstinner commented on Oct 12, 2023

blackliner commented on May 23, 2024

vstinner commented on May 23, 2024

blackliner commented on May 23, 2024

vstinner commented on May 23, 2024

wwarriner commented on Apr 25, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions