Repository navigation
executor.map hangs at seemly random iteration when iterable is large #114948
Description
Activity
Is this reproducible without
numpy?np.arangeis simple enough but it would be nice to rule out all the possible externel dependencies in the example.Is this reproducible without
numpy?np.arangeis simple enough but it would be nice to rule out all the possible externel dependencies in the example.Yes, same behavior when using
range.Win10, main, Command Prompt, with
range: I get multiple duplicate tracebacks.File "C:\Programs\Python313\Lib\multiprocessing\spawn.py", line 164, in get_preparation_data _check_not_importing_main() ~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "C:\Programs\Python313\Lib\multiprocessing\spawn.py", line 140, in _check_not_importing_main raise RuntimeError(''' ...<16 lines>... ''') File "C:\Programs\Python313\Lib\multiprocessing\spawn.py", line 140, in _check_not_importing_main raise RuntimeError(''' ...<16 lines>... ''') RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase. This probably means that you are not using fork to start your child processes and you have forgotten to use the proper idiom in the main module: if __name__ == '__main__': ...With the needed (on Windows) clause added, it prints 0 to 12 on separate lines and hangs. ^C and I get 12 interlaced tracebacks with some of the following lines
File "C:\Programs\Python313\Lib\multiprocessing\process.py", line 314, in _bootstrap self.run() File "C:\Programs\Python313\Lib\multiprocessing\process.py", line 108, in run self._target(*self._args, **self._kwargs) File "C:\Programs\Python313\Lib\concurrent\futures\process.py", line 251, in _process_worker call_item = call_queue.get(block=True) ~~~~~~~~~~~~~~^^^^^^^^^^^^ File "C:\Programs\Python313\Lib\multiprocessing\queues.py", line 102, in get with self._rlock: res = self._recv_bytes() File "C:\Programs\Python313\Lib\multiprocessing\queues.py", line 103, in get res = self._recv_bytes() File "C:\Programs\Python313\Lib\multiprocessing\synchronize.py", line 95, in __enter__ return self._semlock.__enter__() KeyboardInterruptThis does not hang infinitely, it just takes a long time. (~1min on my comupter).
This is the main process timeline for a "normal"
mapoperation, where the input isrange(100).The MainThread is doing the
map, whichsubmitthe work item one by one to a dict. In eachsubmit, two thing happen:- a
wakeupsignal is sent toself.thread_wakeupqueue - a
WorkItem's id is added toself._work_idsqueue
During the first call of
submit, an executor manager thread is brought up (Thread-1in the graph), which is in charge of actually distributing the tasks.It will get the
WorkItems fromself._work_idsandputthem in a process queue that has a size ofMAX_WORKERS + 1for the process workers. Because the queue has a maximum size, the manager thread will not be able to put all the tasks in it.When the task queue is full, the manager waits for a broken or wakeup signal in
wait_result_broken_or_wakeup, and this is where the issue happened.In this function, it needs to read all the pending
wakeupsignals inself.thread_wakeupqueue mentioned above. Remember, thesubmitwill not be blocked, it's not lazily evaluated, so each task sends a wakeup signal. This function needs to process ALL of them to be able to send moreWorkItemsto the worker process.Each
poll&recvtakes about 50 us on my computer:
So if you have 1e6 tasks like in the example, you'll need 50s to process all the wake up signals before you can send out more
WorkItems to the process workers.So it's not a real dead lock, it just takes a long time. However, I do agree that if we can do something about it, that would be nice. But it won't make this go any faster even with lazy evaluation - it will make this smoother, as each wakeup signal always need to take 50us and it needs to be processed anyway (but it can be parallelized so if the task itself takes more than that, it should be ignorable if it's not blocking the task distribution).
- a
@gaogaotiantian Thanks for the profiling, please see #114975 for an attempt to improve the large iterator use case
Ah okay this is a false alarm. What you should do is to specify a
chunksizeto batchify you data.The docs says:
For very long iterables, using a large value for chunksize can significantly improve performance compared to the default size of 1
executor.map(print, np.arange(1000000), chunksize=10000)
should just solve the issue.
Reacted by Timothy Burke
Bug report
Bug description:
Minimum working example of issue:
Each time this is run it seems to hang at a different iteration with no apparent CPU or memory usage. May be related to #74028
CPython versions tested on:
3.9
Operating systems tested on:
Linux
Linked PRs