Dramatic slowdown with random.random on free-threading build. #118392
Description
This slowdown was found with one of my favorite benchmarks, which is calculating the pi value with the Monte Carlo method.
import os
import random
import time
from threading import Thread
def monte_carlo_pi_part(n: int, idx: int, results: list[int]) -> None:
count = 0
for i in range(n):
x = random.random()
y = random.random()
if x*x + y*y <= 1:
count += 1
results[idx] = count
n = 10000
threads = []
num_threads = 100
results = [0] * num_threads
a = time.time()
for i in range(num_threads):
t = Thread(target=monte_carlo_pi_part, args=(n, i, results))
t.start()
threads.append(t)
while threads:
t = threads.pop()
t.join()
b = time.time()
print(sum(results) / (n * num_threads) * 4)
print(b-a)Acquiring critical sections for random methods causes this slowdown.
Removing @critical_section from the method, which uses genrand_uint32 and then updating genrand_uint32 to use atomic operation makes the performance acceptable.
| Build | Elapsed | PI |
|---|---|---|
| Default (with specialization) | 0.16528010368347168 | 3.144508 |
| Free-threading (with no specialization) | 0.548654317855835 | 3.1421 |
| Free-threading with my patch (with no specialization) | 0.2606849670410156 | 3.141108 |
Activity
corona10 commentedon Apr 29, 2024
cc @colesbury
pythongh-118392: Update genrand_uint32 to use atomic operation
colesbury commentedon Apr 29, 2024
It'd be great to have the default random number generator be safe and efficient when called from multiple threads, but I think that's a bigger change that requires more discussion and maybe a PEP.
In the meantime, I think the advice should be to use individual
Randomobjects per-thread. In other words, the program should be written as:I think that you also would need #118112 (or deferred reference counting) for the program to be efficient, or otherwise the reference count contention is going to be a bottleneck.
corona10 commentedon Apr 29, 2024
Yeah, it works. so it should be included somewhere in the migration guide.
corona10 commentedon Apr 29, 2024
#118393 (comment)
pythongh-118392: Add warning about random.random for multi thread app
colesbury commentedon Apr 29, 2024
We could also consider adopting a strategy like Go does, at least for the free-threaded build:
seed()has not been called, use a fast per-thread RNG for the global callsseed()is called, switch to the shared state RNG to preserve old behavior.gh-118392: Add note about random.random for multi thread app (gh-118396)
pythongh-118392: Add note about random.random for multi thread app (p…
pythongh-118392: Add note about random.random for multi thread app (p…