Repository navigation
os.cpu_count is problematic on sparc/solaris #73451
Description
Activity
phantal commented
on Jan 13, 2017 phantalmannequinMannequinAuthorI'm attempting to build python 3.6.0 on sparc/solaris 10. After the initial configure/compile complete I ran "make test" and I see:
$ make test running build running build_ext (...) running build_scripts copying and adjusting (...) changing mode of (...) renaming (...) (...) Run tests in parallel using 258 child processes
I'm fairly sure the issue stems from the fact that each core on the machine has 8 "threads" and there's 32 cores (for a total of 256 virtual cores).
Each core can execute 8 parallel tasks only in very specific circumstances. It's intended for use by things like lapack/atlas where you might be doing many computations on the same set of data.
Outside of these more restricted circumstances each core can only handle 2 parallel tasks (or so I gathered from the documentation), so at best this machine could handle 64 backgrounded jobs though I normally restrict my builds to the actual core count or less.
The most common way to get a "realistic" core count on these machines from shell scripts is:
$ core_count=`kstat -m cpu_info | grep core_id | sort -u | wc -l`... though I'm not sure how the test suite is determining the core count. I didn't see any mention of "kstat" anywhere.
phantal commented
on Jan 13, 2017 phantalmannequinMannequinAuthorI forgot to mention, this wasn't an issue in 3.5.1 though I never did check how many jobs it was using.
I ran into other issues building that version and moved to a newer version because at least one of them (logging test race condition) was fixed after 3.5.1.
phantal commented
on Jan 13, 2017 phantalmannequinMannequinAuthorThis is odd. I just went back and re-ran 3.5.1 to see how many cores and it's having the same problem now. So, scratch that last coment.
You don't know it, but you are actually reporting a possible sparc/Solaris specific (I think) bug against os.cpu_count. There was some discussion about how cpu_count might be problematic in this regard.
It doesn't cause any real problem with the tests, though. I routinely run with -j40 on my 2 cpu test box because the test run completes faster that way due to the way many tests spend time waiting for various things.
- changed the title
[-]test suite is attempting to spawn 258 child processes to run tests[/-][+]os.cpu_count is problematic on sparc/solaris[/+]on Jan 13, 2017 phantal commented
on Jan 13, 2017 phantalmannequinMannequinAuthorIt doesn't cause any real problem with the tests, though. I routinely run with -j40 on my 2 cpu test box because the test run completes faster that way due to the way many tests spend time waiting for various things.
In my case it did because it caused enough file descriptors to be allocated that it hit the cap for max open file handles.
As per the current documentation,
os.cpu_count()should return "number of logical CPUs in the system", and so returning 256 on a 32 core machine with 8 threads is correct.We've previously hit the issue of hitting the cap on max open file handles as well and solved it by running the test suite with
ulimit -n 16384.os.cpu_count is problematic on sparc/solaris
The issue title is misleading. os.cpu_count() is correct. The problem is more that the default file descriptor is too low for a machine with 256 logical CPUs ("threads") when running
./python -m test -j0which spawnsos.process_cpu_count() + 2worker processes.You can specify the number of worker processes:
./python -m test -j20, or you can increase the file descriptor limit.Is there anything to do on the Python side? Should regrtest limit the number of worker processes based on the current file descriptor limit? I have no idea how to compute a number of processes based on a maximum number of file descriptors.
Each core can execute 8 parallel tasks only in very specific circumstances. It's intended for use by things like lapack/atlas where you might be doing many computations on the same set of data.
Reading it again, I see what you mean - SPARC CPUs are not able to execute all those 8 threads at the same time. That said, Intel hyperthreaded cores are also presented as 2 logical cores, and they cannot do certain operations simultaneously (although SPARC is definitely more restricted in this).
Is there anything to do on the Python side? Should regrtest limit the number of worker processes based on the current file descriptor limit? I have no idea how to compute a number of processes based on a maximum number of file descriptors.
That would be very hard to do, considering that tests can then open additional files. I don't think that is really feasible when simply adjusting the
ulimitor the possibility of running the test suite with-jXexists. (though OP might see it differently)The bare minimum which can be easily done is to emit a warning if the FD limit looks too small compared to the -jN argument. Maybe with instructions on how to fix the issue (reduce -jN value or increase FD limit).
That is true. In our case, IIRC, we saw occasional issues when building with fd limit of 4096 with 514 worker processes. So I'd propose warning the user when the fd limit is less than 10 times the worker count?
On Linux, the default FD limit is 1024 but I never got such issue. Well, my laptop "only" has 12 logical CPUs. Which process is impacted by the issue? The main process which spawns and manages the 514 worker processes? Does Python even need a single FD for worker process?
Sorry for such a long delay.
I looked into this and apparently it wasn't the test suite, but pgo phase when this was an issue (back when pgo was running the entire test suite). Also, it seems that back then the default limit on our build machines might have been lower.
Anyway, when I change the limit a lot all the way to 512 and run the test suite, I see the following error:
Warning -- regrtest worker thread failed: Traceback (most recent call last): File "/builds/python37/Python-3.7.17/Lib/test/libregrtest/runtest_mp.py", line 284, in run mp_result = self._runtest(test_name) File "/builds/python37/Python-3.7.17/Lib/test/libregrtest/runtest_mp.py", line 249, in _runtest retcode, stdout, stderr = self._run_process(test_name) File "/builds/python37/Python-3.7.17/Lib/test/libregrtest/runtest_mp.py", line 194, in _run_process popen = run_test_in_subprocess(test_name, self.ns) File "/builds/python37/Python-3.7.17/Lib/test/libregrtest/runtest_mp.py", line 74, in run_test_in_subprocess **kw) File "/builds/python37/Python-3.7.17/Lib/subprocess.py", line 800, in __init__ restore_signals, start_new_session) File "/builds/python37/Python-3.7.17/Lib/subprocess.py", line 1457, in _execute_child errpipe_read, errpipe_write = os.pipe() OSError: [Errno 24] Too many open files Kill <TestWorkerProcess #1 running test=test_support pid=44123 time=8.7 sec> process group Kill <TestWorkerProcess #2 running test=test_fork1 pid=44801 time=4.4 sec> process group ... Kill <TestWorkerProcess #175 running test=test_pickletools pid=45171 time=2.3 sec> process group 0:00:14 load avg: 33.55 Waiting for <TestWorkerProcess #95 running test=test_posix pid=45022 time=4.5 sec> thread for 1.1 sec == Tests result: FAILURE ==which looks like the main process, but I guess that e.g. subprocess tests might fail as well?
But even with limit of 512 it doesn't happen always (I guess that some tests finish very fast and free the resources before the limit is reached)
SPARC/Solaris is unsupported per PEP-11.
I'll close the bug.
Like other issues for unsupported platforms, it'll be listed on the GitHub project for anyone who wants to support the platform unofficially or bring support back.
I encourage such people to add themselves to the Experts list so we know whom to ping.
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields: