Repository navigation
2.8.1 breaks on read only filesystems #905
Description
We updated granian from 2.7.9 to 2.8.1 and it did not come up in our k8s setup. We tracked that down to a) our read only filesystems in k8s (inclusing tmp...) and b) the change in http://github.com/emmett-framework/granian/pull/897
The PR basically changed the default multiprocessing mode in our case from spawn to forkserver:
https://github.com/emmett-framework/granian/pull/897/changes#diff-1ff4544dd6318231db4ca945525391966ce48ed2701c3e159fd648283dd23f18L49-R53
The default start method is forkserver
# Get a python REPL on the hardned container with the read only filesystem:
# kubectl debug -it <currently-still-running-pod> --image=alpine --target=<container> --profile=general --namespace <namespace>
/ # ps aux
/ # chroot /proc/12/root python3.14
>>> import multiprocessing
/usr/lib/python3.14/_pyrepl/simple_interact.py:156: UserWarning: failed to open the history file for writing: [Errno 30] Read-only file system: '/root/.python_history'
warnings.warn(f"failed to open the history file for writing: {e}")
>>> multiprocessing.get_start_method()
'forkserver'We start up granien from a dockerfile with
WORKDIR /workdir
ENV PATH="/opt/app/venv/bin:$PATH"
# Granian specific environment variables
ENV GRANIAN_INTERFACE="asgi"
ENV GRANIAN_HOST="0.0.0.0"
ENV GRANIAN_PORT="8009"
ENV GRANIAN_LOG_ACCESS_ENABLED=true
ENV GRANIAN_LOG_LEVEL="info"
ENV GRANIAN_WORKERS="3"
ENV GRANIAN_THREADS="1"
ENV GRANIAN_BACKLOG="1024"
# Proxy configuration
# Granian uses trust-proxy/proxy-headers logic
ENV GRANIAN_TRUST_PROXY="true"
ENTRYPOINT ["granian", "--factory", "module.app:create_app"]The granian startup then ended up with the following error deep in multiprocessing code:
[INFO] Starting granian (main PID: 1)
[INFO] Listening at: http://0.0.0.0:8009
Traceback (most recent call last):
File "/opt/app/venv/bin/granian", line 10, in <module>
sys.exit(cli.entrypoint())
~~~~~~~~~~~~~~^^
File "/opt/app/venv/lib/python3.14/site-packages/granian/cli.py", line 606, in entrypoint
cli(auto_envvar_prefix='GRANIAN')
~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app/venv/lib/python3.14/site-packages/click/core.py", line 1524, in __call__
return self.main(*args, **kwargs)
~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/opt/app/venv/lib/python3.14/site-packages/click/core.py", line 1445, in main
rv = self.invoke(ctx)
File "/opt/app/venv/lib/python3.14/site-packages/click/core.py", line 1308, in invoke
return ctx.invoke(self.callback, **ctx.params)
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app/venv/lib/python3.14/site-packages/click/core.py", line 877, in invoke
return callback(*args, **kwargs)
File "/opt/app/venv/lib/python3.14/site-packages/granian/cli.py", line 600, in cli
server.serve()
~~~~~~~~~~~~^^
File "/opt/app/venv/lib/python3.14/site-packages/granian/server/mp.py", line 513, in serve
super().serve(spawn_target, target_loader, wrap_loader)
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app/venv/lib/python3.14/site-packages/granian/server/common.py", line 760, in serve
serve_method(spawn_target, target_loader)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app/venv/lib/python3.14/site-packages/granian/server/common.py", line 588, in _serve
self.startup(spawn_target, target_loader)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app/venv/lib/python3.14/site-packages/granian/server/common.py", line 494, in startup
self._spawn_workers(spawn_target, target_loader)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app/venv/lib/python3.14/site-packages/granian/server/common.py", line 346, in _spawn_workers
wrk.start()
~~~~~~~~~^^
File "/opt/app/venv/lib/python3.14/site-packages/granian/server/common.py", line 72, in start
self.inner.start()
~~~~~~~~~~~~~~~~^^
File "/usr/lib/python3.14/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
~~~~~~~~~~~^^^^^^
File "/usr/lib/python3.14/multiprocessing/context.py", line 306, in _Popen
return Popen(process_obj)
File "/usr/lib/python3.14/multiprocessing/popen_forkserver.py", line 35, in __init__
super().__init__(process_obj)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
File "/usr/lib/python3.14/multiprocessing/popen_fork.py", line 20, in __init__
self._launch(process_obj)
~~~~~~~~~~~~^^^^^^^^^^^^^
File "/usr/lib/python3.14/multiprocessing/popen_forkserver.py", line 51, in _launch
self.sentinel, w = forkserver.connect_to_new_process(self._fds)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
File "/usr/lib/python3.14/multiprocessing/forkserver.py", line 89, in connect_to_new_process
self.ensure_running()
~~~~~~~~~~~~~~~~~~~^^
File "/usr/lib/python3.14/multiprocessing/forkserver.py", line 166, in ensure_running
address = connection.arbitrary_address('AF_UNIX')
File "/usr/lib/python3.14/multiprocessing/connection.py", line 82, in arbitrary_address
return os.path.join(util.get_temp_dir(),
~~~~~~~~~~~~~~~~~^^
File "/usr/lib/python3.14/multiprocessing/util.py", line 216, in get_temp_dir
base_tempdir = _get_base_temp_dir(tempfile)
File "/usr/lib/python3.14/multiprocessing/util.py", line 171, in _get_base_temp_dir
base_tempdir = tempfile.gettempdir()
File "/usr/lib/python3.14/tempfile.py", line 312, in gettempdir
return _os.fsdecode(_gettempdir())
~~~~~~~~~~~^^
File "/usr/lib/python3.14/tempfile.py", line 305, in _gettempdir
tempdir = _get_default_tempdir()
File "/usr/lib/python3.14/tempfile.py", line 222, in _get_default_tempdir
raise FileNotFoundError(_errno.ENOENT,
"No usable temporary directory found in %s" %
dirlist)
FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/workdir']
It would be nice to have a knob to override the multiprocessing mode directly from granian or if granian could detect such a read only filesystem and choses spawn then...
Activity
Always patch `multiprocessing` spawn method (close emmett-framework#905)