Skip to content

os.cpu_count() returns wrong number of processors on system with > 64 logical processors #74766

Closed
Closed
os.cpu_count() returns wrong number of processors on system with > 64 logical processors#74766
Labels
3.7 (EOL)end of lifeOS-windowstype-bugAn unexpected behavior, bug, or error

Description

@robbuckley
robbuckley
mannequin
opened on Jun 6, 2017
BPO 30581
Nosy @pfmoore, @pitrou, @vstinner, @giampaolo, @tjguk, @zware, @zooba, @tzickel, @crwilcox, @robbuckley
PRs
  • bpo-30581: Windows: os.cpu_count() returns wrong number of processors #2934
  • bpo-30581: Backport to 3.6: Windows: os.cpu_count() returns wrong number of processors #3267
    		•
    Files
  • ci.txt
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2017-09-01.19:29:13.752>
created_at = <Date 2017-06-06.12:49:39.499>
labels = ['type-bug', '3.7', 'OS-windows']
title = 'os.cpu_count() returns wrong number of processors on system with > 64 logical processors'
updated_at = <Date 2017-12-01.09:45:38.001>
user = 'https://github.com/robbuckley'

    bugs.python.org fields:

    activity = <Date 2017-12-01.09:45:38.001>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2017-09-01.19:29:13.752>
closer = 'pitrou'
components = ['Windows']
creation = <Date 2017-06-06.12:49:39.499>
creator = 'robbuckley'
dependencies = []
files = ['46928']
hgrepos = []
issue_num = 30581
keywords = []
message_count = 18.0
messages = ['295254', '295255', '295258', '295365', '295377', '299462', '299476', '299477', '299480', '301017', '301018', '301033', '301146', '301147', '301150', '301151', '307374', '307376']
nosy_count = 10.0
nosy_names = ['paul.moore', 'pitrou', 'vstinner', 'giampaolo.rodola', 'tim.golden', 'zach.ware', 'steve.dower', 'tzickel', 'crwilcox', 'robbuckley']
pr_nums = ['2934', '3267']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue30581'
versions = ['Python 3.6', 'Python 3.7']

    Activity

    1. robbuckley commented on Jun 6, 2017

      @robbuckley
      robbuckleymannequin
      on Jun 6, 2017
      MannequinAuthor

      os.cpu_count() seems to report incorrect values on windows systems with >64 logical processors

      tried it on 2 similar systems, both running windows 7 / 10 with python 3.6.1 64bit (anaconda):

      platform1 - 2x Xeon E5-2698v4. 20 cores/CPU = total 80 logical cpus with hyperthreading
      platform2 - 2x Xeon E5-2697v3. 14 cores/CPU = total 56 logical cpus with hyperthreading

      os.cpu_count() reports 40 cores on platform1 and 56 on platform2

      I would expect 80 and 56 respectively.

      I suppose this is because the windows api call used is not aware of processor groups, and reports only the number of processors in the current processor group ( eg GetSystemInfo vs GetMaximumProcessorCount )

    2. robbuckley
      added
      OS-windows
      type-bugAn unexpected behavior, bug, or error
      on Jun 6, 2017

    3. vstinner commented on Jun 6, 2017

      @vstinner
      vstinner
      on Jun 6, 2017
      Member

      On Windows, os.cpu_count() is currently implemented with:

      "GetSystemInfo(&sysinfo); return sysinfo.dwNumberOfProcessors;"

      https://msdn.microsoft.com/en-us/library/windows/desktop/ms724958(v=vs.85).aspx

      It seems to return the number of *logical* CPUs:

      """
      dwNumberOfProcessors

      The number of logical processors in the current group.

      Note: For information about the physical processors shared by logical processors, call GetLogicalProcessorInformationEx with the RelationshipType parameter set to RelationProcessorPackage (3).
      """

      It seems like you have two physical CPU packages. Maybe the function only returns infos from the first package?

    4. robbuckley commented on Jun 6, 2017

      @robbuckley
      robbuckleymannequin
      on Jun 6, 2017
      MannequinAuthor

      yes, i believe its reporting the number of processors in the current group only, not across all groups.

      attached output of windows sysinternals/coreinfo showing 2 processor groups

      see giampaolo/psutil#771 for some further disucssion of this topic

      the maintainer of psutil asked me to raise this bug, also had a quick check on #python IRC. Its my first bug on bugs.python.org so if you need more info just let me know

    5. crwilcox commented on Jun 7, 2017

      @crwilcox
      crwilcoxmannequin
      on Jun 7, 2017
      Mannequin

      I am going to work on this if no one else has started.

    6. giampaolo commented on Jun 7, 2017

      @giampaolo
      giampaolo
      on Jun 7, 2017
      Contributor

      Nobody has AFAIK.

    7. pitrou
      added
      3.7 (EOL)end of life
      on Jun 29, 2017

    8. giampaolo commented on Jul 29, 2017

      @giampaolo
      giampaolo
      on Jul 29, 2017
      Contributor

      MS documentation is not clear on what function should be used as there are many returning different values. Here it is being suggested to use GetLogicalProcessorInformationEx:
      https://stackoverflow.com/questions/31209256/reliable-way-to-programmatically-get-the-number-of-cores-on-windows

    9. crwilcox commented on Jul 29, 2017

      @crwilcox
      crwilcoxmannequin
      on Jul 29, 2017
      Mannequin

      I agree that the MS Docs for this are a bit confusing. I ended up reaching out to the guy who authored the GetMaximumProcessorCount function. I had also written an implementation that iterated over GetProcessorInformationEx and he advised against it.

      One of the things that makes this interesting is that in 32 bit processes (wow64) your processor is simulated to fit in the confines of that old system. This method will only report 32 cores under 32 bit as that is all the program can access in 32 bit mode.

    10. giampaolo commented on Jul 29, 2017

      @giampaolo
      giampaolo
      on Jul 29, 2017
      Contributor

      About GetMaximumProcessorCount, MS doc states that it returns the "maximum number of logical processors that a processor group or the system can have", so maybe it also includes "empty" CPU sockets.

      GetActiveProcessorCount, on the other hand, returns "the number of active processors in a processor group or in the system", which adds even more confusion.

    11. crwilcox commented on Jul 29, 2017

      @crwilcox
      crwilcoxmannequin
      on Jul 29, 2017
      Mannequin

      I was reviewing the docs for the os module and cpu_count should always return the number of cpus on the system, not the usable CPUs. GetMaximumProcessorCount returns a simulated count in WoW64. I have reached back out to the Windows API dev and will see if GetLogicalProcessorInformationEx will allow us to do this. He had thought that my solution that way had other limitations under WoW64.

    12. pitrou commented on Aug 30, 2017

      @pitrou
      pitrou
      on Aug 30, 2017
      Member

      New changeset c67bae0 by Antoine Pitrou (Christopher Wilcox) in branch 'master':
      bpo-30581: Windows: os.cpu_count() returns wrong number of processors (bpo-2934)
      c67bae0

    13. pitrou commented on Aug 30, 2017

      @pitrou
      pitrou
      on Aug 30, 2017
      Member

      Fixed. Someone might backport this to 3.6 if they want.

    14. pitrou
      closed this as completedon Aug 30, 2017

    15. vstinner commented on Aug 30, 2017

      @vstinner
      vstinner
      on Aug 30, 2017
      Member

      I reopen the issue to backport the bugfix to 3.6.

    16. vstinner
      reopened this on Aug 30, 2017

    17. pitrou commented on Sep 1, 2017

      @pitrou
      pitrou
      on Sep 1, 2017
      Member

      New changeset 58521fd by Antoine Pitrou (Christopher Wilcox) in branch '3.6':
      bpo-30581: Windows: os.cpu_count() returns wrong number of processors (bpo-2934) (bpo-3267)
      58521fd

    18. pitrou commented on Sep 1, 2017

      @pitrou
      pitrou
      on Sep 1, 2017
      Member

      Backport merged. Thanks Chris!

    19. pitrou
      closed this as completedon Sep 1, 2017

    20. tzickel commented on Sep 1, 2017

      @tzickel
      tzickelmannequin
      on Sep 1, 2017
      Mannequin

      One should be careful with this modification because of the Windows definition of process groups.

      For example, if multi-threaded code thinks that by reading the value of the new os.cpu_count() it can use all the cores returned, by default it cannot as in windows processes by default can run only in a single process group (how it worked before).

      We can see such code builtin python stdlib itself:

      cpython/Lib/concurrent/futures/thread.py

      Line 102 in bc61315

      max_workers = (os.cpu_count() or 1) * 5

      I think even .NET still uses the old way that python did until now:
      https://github.com/dotnet/corefx/blob/aaaffdf7b8330846f6832f43700fbcc060460c9f/src/System.Runtime.Extensions/src/System/Environment.Windows.cs#L71

      Although some of this stuff is used in code for python multiprocess code which that might actually get a boost (since different process can get scheduled to different groups)

      https://msdn.microsoft.com/en-us/library/windows/desktop/dd405503(v=vs.85).aspx

    21. pitrou commented on Sep 1, 2017

      @pitrou
      pitrou
      on Sep 1, 2017
      Member

      os.cpu_count() is specified to return the total number of processors, not the number of usable processors. See e.g. https://bugs.python.org/issue26692

    22. robbuckley commented on Dec 1, 2017

      @robbuckley
      robbuckleymannequin
      on Dec 1, 2017
      MannequinAuthor

      hi,
      as the reporter i just want to say this is working for me with 3.6.3.

      Regarding https://bugs.python.org/issue30581#msg301150, I take your point that a lot of multiprocessing using the standard libraries may not benefit, as processes may be restricted to the processor group of the parent process (python).

      For my use case it works well: I launch a queue of blocking jobs, using a thread pool. Each thread launches 1 jobsubprocess.subprocess.run(), where the thread pool size is equal to number of processors reported by os.cpu_count(). Since the OS controls the scheduling in this case, it works perfectly well with 2 processor groups.

      thanks :-)

    23. pitrou commented on Dec 1, 2017

      @pitrou
      pitrou
      on Dec 1, 2017
      Member

      Thanks for the heads up Rob!

    24. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.7 (EOL)end of lifeOS-windowstype-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions