Repository navigation

gh-64192: Make `imap()`/`imap_unordered()` in `multiprocessing.pool` actually lazy - #136871

Merged

encukou merged 35 commits into

python:mainfrom

obaltian:feature/add-buffersize-to-multiprocessing

Jul 29, 2026

Merged

gh-64192: Make `imap()`/`imap_unordered()` in `multiprocessing.pool` actually lazy#136871
encukou merged 35 commits into
python:mainfrom
obaltian:feature/add-buffersize-to-multiprocessing

Commits on Aug 14, 2025
- draft: impl lazy input consumption in mp.Pool.imap(_unordered)
  
  Oleksandr Baltian
  authored and
  obaltian
  committed
  ccd0bbc
- Use semaphore to synchronize threads
  
  Oleksandr Baltian
  authored and
  obaltian
  committed
  002ef46
- Update buffersize behavior to match concurrent.futures.Executor behavior
  
  Oleksandr Baltian
  authored and
  obaltian
  committed
  6e0bc58
- Release all `buffersize_lock` obj from the parent thread when terminate
  
  Oleksandr Baltian
  authored and
  obaltian
  committed
  62b2b6a
- Add 2 basic `ThreadPool.imap()` tests w/ and w/o buffersize
  
  Oleksandr Baltian
  authored and
  obaltian
  committed
  0b6ba41
- Fix accidental swap in imports
  
  Oleksandr Baltian
  authored and
  obaltian
  committed
  aade15e
- clear Pool._taskqueue_buffersize_semaphores safely
  
  Oleksandr Baltian
  authored and
  obaltian
  committed
  fb38a72
- Slightly optimize Pool._taskqueue_buffersize_semaphores terminate
  
  Oleksandr Baltian
  authored and
  obaltian
  committed
  6ef488b
- Rename `Pool.imap()` buffersize-related tests
  
  Oleksandr Baltian
  authored and
  obaltian
  committed
  1716725
- Fix typo in `IMapIterator.__init__()`
  
  Oleksandr Baltian
  authored and
  obaltian
  committed
  9b43cd0
- Add tests for buffersize combinations with other kwargs
  
  Oleksandr Baltian
  authored and
  obaltian
  committed
  2d89341
- Remove if-branch in `_terminate_pool`
  
  Oleksandr Baltian
  authored and
  obaltian
  committed
  9ab2705
- Add more edge-case tests for `imap` and `imap_unodered`
  
  Oleksandr Baltian
  authored and
  obaltian
  committed
  a955003
- Split inf iterable test for `imap` and `imap_unordered`
  
  Oleksandr Baltian
  authored and
  obaltian
  committed
  80efd6e
- Add doc for `buffersize` argument of `imap` and `imap_unordered`
  
  Oleksandr Baltian
  authored and
  obaltian
  committed
  83d6930
- add *versionadded* for `imap_unordered`
  
  Oleksandr Baltian
  authored and
  obaltian
  committed
  995ad8c
- Remove ambiguity in `buffersize` description.
  
  Oleksandr Baltian
  authored and
  obaltian
  committed
  3b6ad65
- Set *versionadded* as next in docs
  
  Oleksandr Baltian
  authored and
  obaltian
  committed
  c941c16
- Add whatsnew entry
  
  Oleksandr Baltian
  authored and
  obaltian
  committed
  d09e891
- Fix aggreed comments on code formatting/minor refactoring
  
  Oleksandr Baltian
  authored and
  obaltian
  committed
  9c6d89d
- Remove `imap` and `imap_unordered` body code duplication
  
  Oleksandr Baltian
  authored and
  obaltian
  committed
  4550a01
Commits on Aug 31, 2025
- Merge branch 'main' into feature/add-buffersize-to-multiprocessing
  
  obaltian
  authored and
  Oleksandr Baltian
  committed
  77bde4d
Commits on Sep 3, 2025
- Merge branch 'main' into feature/add-buffersize-to-multiprocessing
  obaltian
  authored
  aec39fc
Commits on Jul 17, 2026
- Merge remote-tracking branch 'origin/main' into feature/add-buffersize-to-multiprocessing
  obaltian
  committed
  a469f2d
- Remove extra import
  obaltian
  committed
  bcda6e7
Commits on Jul 18, 2026
- Update Lib/multiprocessing/pool.py
  
  obaltian
  and
  encukou
  authored
  1441eaa
- Update Doc/library/multiprocessing.rst
  
  obaltian
  and
  encukou
  authored
  6d61bd1
- Apply suggestion from @encukou
  
  obaltian
  and
  encukou
  authored
  0e43d9c
- Make `buffersize` argument kwarg-only
  Sasha Baltian
  committed
  4ed84bc
- Use set for `._taskqueue_buffersize_semaphores`
  Sasha Baltian
  committed
  f7aeb2e
- Make `buffersize` arg check inline
  Sasha Baltian
  committed
  78b1f58
- Simplify `.imap()` and `.imap_undered()` tests
  Sasha Baltian
  committed
  49b03d9
- Fix iterator instantiation in in `.imap()`
  Sasha Baltian
  committed
  455b748
- Mark kw-only arg in `imap`/`imap_unordered` in doc
  Sasha Baltian
  committed
  56eaf3f
- Merge branch 'main' into feature/add-buffersize-to-multiprocessing
  obaltian
  authored
  ea540bf

Commits

Commits on Aug 14, 2025

Commits on Aug 31, 2025

Commits on Sep 3, 2025

Commits on Jul 17, 2026

Commits on Jul 18, 2026