Repository navigation
Minor InterpreterPoolExecutor Issues #133485
Description
Activity
I ran into this with the following example:
import argparse import concurrent.futures import random import statistics import time EXECUTORS = { "thread": concurrent.futures.ThreadPoolExecutor, "process": concurrent.futures.ProcessPoolExecutor, "interpreter": concurrent.futures.InterpreterPoolExecutor, } def pi(trials: int) -> float: Ncirc = 0 ran = random.Random() for _ in range(trials): x = ran.uniform(-1, 1) y = ran.uniform(-1, 1) test = x * x + y * y if test <= 1: Ncirc += 1 return 4.0 * (Ncirc / trials) def pi_in_threads(threads: int, trials: int, executor: str) -> float: if threads == 0: return pi(trials) Executor = EXECUTORS[executor] with Executor(max_workers=threads) as executor: return statistics.mean(executor.map(pi, [trials // threads] * threads)) def main() -> None: parser = argparse.ArgumentParser(color=True, suggest_on_error=True) parser.add_argument("--threads", type=int, default=0) parser.add_argument("--trials", type=int, default=10_000_000) parser.add_argument("--executor", choices=EXECUTORS, default="thread") args = parser.parse_args() start = time.monotonic() π = pi_in_threads(args.threads, args.trials, args.executor) stop = time.monotonic() print(f"{args.trials} trials, {π = }, {stop - start:.4} s") if __name__ == "__main__": main()
It's unable to run with
--executor interpreterbecause it hitsoncurrent.futures.interpreter.ExecutionFailed: AttributeError: module '__main__' has no attribute 'pi'. The other two work. I'd assume this might be similar to how ProcessPoolExectuor handles__main__. Good to see there's already an issue, though, just dropping the script in case it's helpful.The
syncobjparameter in the PEP isn't really explained, and it doesn't even seem to be present:Traceback (most recent call last): File "/Users/henryschreiner/git/software/pybind11/concint.py", line 6, in <module> tasks = concurrent.interpreters.create_queue(syncobj=True) TypeError: create() got an unexpected keyword argument 'syncobj'I'm currently just sticking an integer into a queue, and in main interpreter I'm getting
concurrent.interpreters._queues.UNBOUNDout. I was wondering if this was related, but it's not even present AFAICT. I seeunbounditems=in the code, which maybe is a new name? Is the plan to update the PEP, or just rely on CPython docs?Is there some way to copy the item so it can be accessed after the interpreter is destroyed? This is my demo code:
Code:
import concurrent.interpreters import concurrent.futures import contextlib import inspect tasks = concurrent.interpreters.create_queue() with ( contextlib.closing(concurrent.interpreters.create()) as interp1, contextlib.closing(concurrent.interpreters.create()) as interp2 ): with concurrent.futures.ThreadPoolExecutor() as pool: for interp in [interp1, interp2]: interp.prepare_main(tasks=tasks) pool.submit(interp.exec, inspect.cleandoc(""" import time time.sleep(5) tasks.put(1) """ )) tasks.put(2) print(tasks.get_nowait(), tasks.get_nowait(), tasks.get_nowait())
Indenting the final two lines fixes it, but curious if there's a way to get copies so I don't have to worry about the lifetimes, especially for things like integers.
- added a commit that references this issue
on Jun 18, 2025 It's unable to run with --executor interpreter because it hits oncurrent.futures.interpreter.ExecutionFailed: AttributeError: module 'main' has no attribute 'pi'. The other two work. I'd assume this might be similar to how ProcessPoolExectuor handles main. Good to see there's already an issue, though, just dropping the script in case it's helpful.
FYI, I just landed the key change here, which will be in beta 4. It should take care of the discrepancy between
InterpreterPoolExecutorand the others.The syncobj parameter in the PEP isn't really explained, and it doesn't even seem to be present:
Yeah, it's gone. I forgot to remove it from the PEP. I'll fix that.
Is there some way to copy the item so it can be accessed after the interpreter is destroyed? This is my demo code:
Objects belong to interpreters and an object cannot outlive the interpreter that created it.
- added a commit that references this issue
on Jun 19, 2025 The issues I knew of have been addressed. If there are any other specific inconsistencies (or bugs) then feel free to open a new issue.
I've updated PEP 734: python/peps#4467.
Metadata
Metadata
Assignees
Labels
Projects
- Status
Bug report
Bug description:
__main__handling isn't quite rightThis has gh-132775 as a prerequisite.
CPython versions tested on:
CPython main branch
Operating systems tested on:
No response
Linked PRs