run_process fails when running from .exe console_script entrypoint on windows

Things to check first

I have searched the existing issues and didn't find my bug already reported there
I have checked that my bug is still present in the latest release

AnyIO version

4.3.0

Python version

3.11.3

What happened?

running:

import anyio.to_process

from fastapi import FastAPI


app = FastAPI()

def sync_example():
    return 1

@app.get("/")
async def example() -> int:
    return await anyio.to_process.run_sync(sync_example)

with uvicorn demo:app fails with:

Traceback (most recent call last):
  File "C:\Users\User\uvicron_demo\Lib\site-packages\anyio\to_process.py", line 163, in run_sync
    await send_raw_command(pickled)
  File "C:\Users\User\uvicron_demo\Lib\site-packages\anyio\to_process.py", line 89, in send_raw_command
    raise retval
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\User\\uvicron_demo\\Scripts\\uvicorn.exe\\__main__.py'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\User\uvicron_demo\Lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 412, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\uvicron_demo\Lib\site-packages\uvicorn\middleware\proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\uvicron_demo\Lib\site-packages\fastapi\applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "C:\Users\User\uvicron_demo\Lib\site-packages\starlette\applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "C:\Users\User\uvicron_demo\Lib\site-packages\starlette\middleware\errors.py", line 186, in __call__
    raise exc
  File "C:\Users\User\uvicron_demo\Lib\site-packages\starlette\middleware\errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "C:\Users\User\uvicron_demo\Lib\site-packages\starlette\middleware\exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "C:\Users\User\uvicron_demo\Lib\site-packages\starlette\_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "C:\Users\User\uvicron_demo\Lib\site-packages\starlette\_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "C:\Users\User\uvicron_demo\Lib\site-packages\starlette\routing.py", line 758, in __call__
    await self.middleware_stack(scope, receive, send)
  File "C:\Users\User\uvicron_demo\Lib\site-packages\starlette\routing.py", line 778, in app
    await route.handle(scope, receive, send)
  File "C:\Users\User\uvicron_demo\Lib\site-packages\starlette\routing.py", line 299, in handle
    await self.app(scope, receive, send)
  File "C:\Users\User\uvicron_demo\Lib\site-packages\starlette\routing.py", line 79, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "C:\Users\User\uvicron_demo\Lib\site-packages\starlette\_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "C:\Users\User\uvicron_demo\Lib\site-packages\starlette\_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "C:\Users\User\uvicron_demo\Lib\site-packages\starlette\routing.py", line 74, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\uvicron_demo\Lib\site-packages\fastapi\routing.py", line 278, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\uvicron_demo\Lib\site-packages\fastapi\routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\demo.py", line 13, in example
    return await anyio.to_process.run_sync(sync_example)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\uvicron_demo\Lib\site-packages\anyio\to_process.py", line 168, in run_sync
    raise BrokenWorkerProcess(
anyio.BrokenWorkerProcess: Error during worker process initialization

How can we reproduce the bug?

see above

Where should it get the interpreter path then?

sys.executable ?

I think the problem is more related to uvicorn rather than to anyio. Try using another asgi web server, for example hypercorn or run you app via main section like this:

if __name__ == '__main__':
    uvicorn.run('app:app', host="0.0.0.0", port=8000)

As stated above, the problem is where to get the path to the interpreter, as sys.executable points to uvicorn.exe.

Then one of the workarounds might be using the following run command:

python -m uvicorn app:app

In this case sys.executable should point to python.exe

Yes, that certainly should work.

One thing that occurred to me: does multiprocessing still work from within a process launched as uvicorn? If so, where does it get the path to the actual Python interpreter?

After digging a while, I saw in multiprocessing/spawn.py that it was doing os.path.join(sys.exec_prefix, 'python.exe'). Probably not a bad choice.

I did some more digging and turns out, I was completely wrong about where the problem occurs. sys.executable was never the culprit. The problem was that it didn't know where to find the __main__ module. What I do there is getattr(sys.modules["__main__"], "__file__", None), but on Windows, that returns C:\...\uvicorn.exe\__main__.py! Even more bizarre is that if you try the same from the module itself, its __file__ attribute is completely fine.

Right, I keep forgetting that the target function doesn't even live in the __main__ module, so there is no real point in trying to import that module. The subprocess end should check if the path to __main__ exists, and only then try to import it.

Alright, this was a one-liner fix in the end.

Sounds cool, I can test it right away.

I'm struggling a bit to write a failing test right now.

I now understand why it only failed on Windows: __main__.__file__ points to a Python script bin/uvicorn on Linux/macOS, but on Windows, it points to an .exe wrapper which cannot be loaded by the Python import machinery. Thus I believe I need to point __main__.__file__ to an existing file that would fail to be imported.

Fixed to_process.run_sync() sometimes failing to initialize on Windows

@adilkhash

@adilkhash feel free to test against the linked PR.

it works! seems like the new release is coming 😉
thanks

looks like something is wrong, if I press Ctrl+C and KeyboardInterrupt is raised the following exception occurs:


Traceback (most recent call last):
  File "C:\...\Lib\asyncio\events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "C:\...\Lib\asyncio\subprocess.py", line 72, in pipe_data_received
    reader.feed_data(data)
  File "C:\...\Lib\asyncio\streams.py", line 477, in feed_data
    assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof

Can you share the script you tested with?

The one that is listed in this issue.
UPD: sorry, looks like it doesn't relate to that issue.

Ok, so are you able to reproduce this other issue? Can you tell me how?

I am not sure if it is related to anyio at all.

Fixed to_process.run_sync() sometimes failing to initialize on Windows (

Bump anyio from 4.4.0 to 4.5.0 in /api (#297)

Bump anyio from 4.4.0 to 4.6.0 (#156)

Bump anyio from 4.5.0 to 4.6.0 in /api (#298)

Navigation Menu

Repository navigation

run_process fails when running from .exe console_script entrypoint on windows #696

Things to check first

AnyIO version

Python version

What happened?

How can we reproduce the bug?

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

run_process fails when running from .exe console_script entrypoint on windows #696

Description

Things to check first

AnyIO version

Python version

What happened?

How can we reproduce the bug?

Activity

agronholm commented on Mar 7, 2024

graingert commented on Mar 7, 2024

adilkhash commented on Jun 7, 2024

agronholm commented on Jun 7, 2024

adilkhash commented on Jun 7, 2024

agronholm commented on Jun 7, 2024

agronholm commented on Jun 7, 2024

agronholm commented on Jun 7, 2024

agronholm commented on Jun 7, 2024

agronholm commented on Jun 7, 2024

agronholm commented on Jun 7, 2024

adilkhash commented on Jun 7, 2024

agronholm commented on Jun 7, 2024

agronholm commented on Jun 7, 2024

agronholm commented on Jun 7, 2024

adilkhash commented on Jun 7, 2024

adilkhash commented on Jun 7, 2024

agronholm commented on Jun 7, 2024

adilkhash commented on Jun 9, 2024

agronholm commented on Jun 9, 2024

adilkhash commented on Jun 9, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions