Repository navigation
Add concurrent.futures.InterpreterPoolExecutor #124694
Description
Activity
- added a commit that references this issue
on Oct 18, 2024 Does the pickling work?I've been trying several things on pickable arguments, but I keep gettingconcurrent.interpreters.NotShareableError: args not shareable. My first thought was this:# /// script # dependencies = ["numpy"] # requires-python = ">=3.14" # /// import numpy as np from concurrent.futures import InterpreterPoolExecutor def identity(*args): return args[0] if __name__ == "__main__": x = np.random.rand(10003) xs = np.array_split(x, 4) with InterpreterPoolExecutor(max_workers=4) as pool: results = pool.map(identity, xs) final = np.concat(results) assert np.all(x == final)
But I get that error. Then I tried explicitly pickling:
# /// script # dependencies = ["numpy"] # requires-python = ">=3.14" # /// import numpy as np import pickle from concurrent.futures import InterpreterPoolExecutor def identity(*args): return pickle.dumps(pickle.loads(args[0])) if __name__ == "__main__": x = np.random.rand(10003) xs = np.array_split(x, 4) xsp = (pickle.dumps(a) for a in xs) with InterpreterPoolExecutor(max_workers=4) as pool: results = pool.map(identity, xsp) final = np.concat([pickle.loads(r) for r in results]) assert np.all(x == final)
But I still get this error:
Exception: AttributeError: module '__main__' has no attribute 'identity' The above exception was the direct cause of the following exception: concurrent.interpreters.NotShareableError: object could not be unpickled The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/Users/henryschreiner/git/scikit-hep/boost-histogram/example.py", line 21, in <module> final = np.concat([pickle.loads(r) for r in results]) ^^^^^^^ File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/futures/_base.py", line 639, in result_iterator yield _result_or_cancel(fs.pop()) ~~~~~~~~~~~~~~~~~^^^^^^^^^^ File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/futures/_base.py", line 311, in _result_or_cancel return fut.result(timeout) ~~~~~~~~~~^^^^^^^^^ File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/futures/_base.py", line 450, in result return self.__get_result() ~~~~~~~~~~~~~~~~~^^ File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/futures/_base.py", line 395, in __get_result raise self._exception File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/futures/thread.py", line 86, in run result = ctx.run(self.task) File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/futures/interpreter.py", line 85, in run return self.interp.call(do_call, self.results, *task) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/interpreters/__init__.py", line 233, in call return self._call(callable, args, kwargs) ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/interpreters/__init__.py", line 215, in _call res, excinfo = _interpreters.call(self._id, callable, args, kwargs, restrict=True) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ concurrent.interpreters.NotShareableError: args not shareableAlso, I'm not sure why the
Exception: AttributeError: module '__main__' has no attribute 'identity'without a traceback is at the top. Other executors are fine with code.I was actually trying to use this in a test for boost-histogram, trying to pass a numpy array in and a boost-histogram object back out, but it fails even with byte strings from pickle above.
Edit: I do think it's an issue with the import like seen at the top, and the traceback is a red herring. The tests version of this list
Exception: ModuleNotFoundError: No module named 'tests'at the top. So it's not able to import the function to run. I thought__main__support was fixed?It seems the subinterpreter's don't inherit the current
sys.path? Using pytest 7'spythonpathsetting doesn't work, they aren't picking up the added test paths.If I put the function inside my package, I get
ImportError: NumPy: dtype is already registeredwhich I think that's occurring when pybind11 is registering my dtype. I've opened an issue pybind/pybind11#5781; if I manually skip over that, it looks like numpy doesn't support subinterpreters yet, so I'd not really be able to do much with it yet.- added a commit that references this issue
on Oct 25, 2025
Metadata
Metadata
Assignees
Labels
Projects
- Status
Feature or enhancement
Proposal:
While we wait on PEP 734, there's nothing blocking us from adding a new executor for multiple interpreters. Doing so would allow people to start trying them out. (I wish I had thought of this for 3.13!)
My planned design:
ThreadPoolExecutorsubmit()ormap()will be pickled (same asProcessPoolExecutor)submit()arg can be a script instead of a callable (a script formap()doesn't make sense since there are no args)__init__()will take a new "shared" arg that corresponds to whatInterpreter.prepare_main()takes in PEP 734, to allow each worker interpreter to share some dataIn the future we could support a more efficient "sharing" scheme than pickle. When (or if) PEP 734 is accepted, we can refactor the executor to use the
interpretersmodule rather than using_interpretersdirectly.Open questions:
ProcessPoolExecutor)?CC @brianquinlan
Has this already been discussed elsewhere?
This is a minor feature, which does not need previous discussion elsewhere
Links to previous discussion of this feature:
No response
Linked PRs