Make pdb asyncio aware

Feature or enhancement

Proposal:

Currently pdb does not know anything about asyncio, even though it's a feature officially supported for a long time. I will add supports for asyncio in roughly 3 steps (does not have to be exact 3 PRs):

pdb will generate some passive info for asyncio. If the user hits a breakpoint inside an asyncio task, there will be a one line message with the current stack indicating which task the user is in. A convenience variable $_asynctask will be added to refer to the current task.
pdb will provide a way for the user to examine all the tasks in the current event loop with a new command. The user can inspect how many tasks are there and their status. Maybe they can even check the frames for each task.
pdb will enable users to switch between tasks, aka letting users to specify "run until I'm in this task". pdb will also possibly provide a way for the users to cancel the task.

In theory, many of the features are already possible with manual code as pdb can execute arbitrary code anyway. However, I believe it's helpful for the users to have a more convenient way to debug their async programs.

This was briefly mentioned in the language summit at PyCon this year, no one objected (maybe because they have other stuff that they were more against :) ).

This should not impact any user experience with the existing code that not involves asyncio.

If you have any other suggestions or objections, please let me know.

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

Linked PRs

pdb will enable users to switch between tasks, aka letting users to specify "run until I'm in this task". pdb will also possibly provide a way for the users to cancel the task.

I have some questions:

How would you implement this?
Will it also switch the current context?
What if the task runs in separate event loop in a different thread?
How would this interact with async generators if any?

Switching is basically wait until in that task. pdb will have a callback for each function call(including entering generator) and we just need to check if the callee is what we need.

Multi-threading is not supported at all in pdb. Anything involves multi-threading is a non-blocker because - well it won't work anyway.

I did not think about async generators specifically, but as far as I understand, it's not that special is it? Maybe we can deal with it when there's a concrete example.

What I'm trying to do now is to have some level of support of asyncio, because have none now (actually I don't think the debuggers out there have support it much either). There will be issues that are difficult to solve, and there might be some things that are not achievable in the short term. There are things that we can't do in a single process single thread sync Python program. We may not be able to build a perfect all-can-do debugger for async, but we should still try.

What I'm trying to do now is to have some level of support of asyncio, because have none now (actually I don't think the debuggers out there have support it much either). There will be issues that are difficult to solve, and there might be some things that are not achievable in the short term. There are things that we can't do in a single process single thread sync Python program. We may not be able to build a perfect all-can-do debugger for async, but we should still try.

I look forward to try a prototype of this!

@kumaraditya303

@kumaraditya303 could you take a quick look at #124367? It's just a draft for now because I need to change all the asyncio related pdb doctests before merging it. It implemented step 1. One thing I'd really like to know is if the information Task-1: <main pending> enough. asyncio tasks have their own repr and of course we can use that in pdb directly, but the repr is really long, including a lot of information. The short version is something we want to show the users every time they stop inside an async task and I think it should be more concise. They can always do a $_asynctask to get the current running task. Do you think there are more critical information that the user will very often want to know from their task? I'm not the most heavy asyncio user in the world and that's the area where I need suggestions from experts.

Task-1: <main pending> isn't so useful, running would be better than pending and it should print the callbacks of the task too.

As for convenience variable, maybe we can have one for the current contextvars used by the task?

I can remember the enter task and set that task to running, that's doable. All the other tasks would be pending, cancelling, cancelled or finished I think.

For the callbacks, one of the issue I had was there could be many and it could be long. Do you think something like Task-1: <main pending> cb=[_run_until_complete_cb()] would be helpful? Again, this is the information we display immediately (and always) when the user hits the breakpoint or steps, only the most important information should be displayed here. In normal cases, it would be

> /home/gaogaotiantian/programs/mycpython/example.py(9)main()
-> breakpoint()

The user can always get the detailed information with $_asynctask:

<Task pending name='Task-1' coro=<main() running at /home/gaogaotiantian/programs/mycpython/example.py:9> cb=[_run_until_complete_cb() at /home/gaogaotiantian/programs/mycpython/Lib/asyncio/base_events.py:182]>

But this line is too long to show in the debugger every time the user stops (it could be even longer).

As for the convenience variable, we have a few ways to show that. The simplest is just

$_asyncvars = $_asynctask.get_context()

Then you can do

$_asyncvars.get(foo)

which is always correct but a bit boring.

Say we have a foo = contextvars.ContextVar('bar', default='default'), what do you want in a debugger? You can have a dedicated command like

(Pdb) asyncvars
<ContextVar name='var' default='default' at 0x7f4442d70ef0>: 'is value'

Or you can have a proxy to support

$_asyncvars[foo] = None
$_asyncvars['var'] = None  # Ignore if there's ambiguity, aka two context vars having the same name

Getting the current context is trivial, the question is what the user need to interface with the context.

gh-121468: Add current asyncio task as a convenience variable in pdb (#…

gh-121468: Fix the doctest failure for asyncio test on pdb (#131258)

pythongh-121468: Add current asyncio task as a convenience variable i…

pythongh-121468: Fix the doctest failure for asyncio test on pdb (pyt…

gh-121468: Ensure PDB cleans up event loop policies after using async…

pythongh-121468: Ensure PDB cleans up event loop policies after using…

gh-121468: Support async breakpoint in pdb (#132576)

Is there any possibility this could be made to be not asyncio specific? There are other concurrency frameworks like trio and the cocotb (which I maintain) that could benefit from this. This could be as simple as providing pdb functionality to switch which coroutine's stack is the stack being operated on by the debugger. We could handle the rest.

Another issue is that this makes breakpoint problematic. Ideally you could use the set_trace_async when in an asyncio Task (or generally just an async function), but set_trace otherwise.

I'm finding one thing hard about set_trace_async. I think it touches a more general problem.

Consider this example:

import asyncio
import pdb

async def foo() -> None:
    await pdb.set_trace_async()
    print("foobar")

asyncio.run(foo())

The problem is that you can only await before you step further:

> /home/bswck/Python/cpython/t.py(5)foo()
-> await pdb.set_trace_async()
(Pdb) l
  1     import asyncio
  2     import pdb
  3  
  4     async def foo() -> None:
  5  ->     await pdb.set_trace_async()
  6         print("foobar")
  7  
  8     asyncio.run(foo())
[EOF]
(Pdb) await asyncio.sleep(0)
> /home/bswck/Python/cpython/t.py(5)foo()
-> await pdb.set_trace_async()
(Pdb) n
> /home/bswck/Python/cpython/t.py(6)foo()
-> print("foobar")
(Pdb) await asyncio.sleep(0)
*** SyntaxError: 'await' outside function
(Pdb) l
  1     import asyncio
  2     import pdb
  3  
  4     async def foo() -> None:
  5         await pdb.set_trace_async()
  6  ->     print("foobar")
  7  
  8     asyncio.run(foo())
[EOF]

The tests don't seem to be exercising that path.

The reason this happens is that we rely on set_trace_async() shim frame to await the ephemeral coroutine from a top-level await instruction, but there is no shim frame as soon as we leave set_trace_async() because of n/s.

As far as I understood, something as straightforward as running an async prompt in a task like in the asyncio REPL is not an option here:

cpython/Lib/asyncio/__main__.py

Line 62 in c117b03

repl_future = self.loop.create_task(coro, context=self.context)

That said, this problem is about generators as well; the problem of await is essentially the same as with yield from.

I was wondering if there could be a way to execute the ephemeral coroutine constructed from a top-level await instruction without having to rely on the shim frame. This would make awaits work everywhere it's needed without even having to worry about awaiting set_trace_async.

My guess was that it could be feasible if generators (and coroutines, and async gens) were mutable. That is, if there was a way to make a generator temporarily yield from a new object (a new generator that can be written in a REPL, let's call it gi_debugging_extra_items) and then, after such temporary iterator is exhausted, use gi_yieldfrom as usual. (gi_yieldfrom cannot be written to because it's a getter that finds the delegation target in the frame stack).

That would roughly work like:

def gen():
    breakpoint()
    yield from range(3)

list(gen())

(Pdb) yield from ["foo"]

->

def __pdb_extra_items():
    yield from ["foo"]

->

caller_frame.f_generator.gi_debugging_extra_items = __pdb_extra_items()

->

list(gen())  # ["foo", 0, 1, 2]

I'm not sure whether that's a good path to go down. Perhaps there could be an easier way to mutate generator-like objects for debugging purposes. How can we lie to _PyGen_yf?

In any case, if that can be done, the problem of await/yield from seems to be fully solved.

I'm not sure whether that's a good path to go down. Perhaps there could be an easier way to mutate generator-like objects for debugging purposes. How can we lie to _PyGen_yf?

In any case, if that can be done, the problem of await/yield from seems to be fully solved.

It's essentially about replacing frame.f_generator.gi_yieldfrom with an itertools.chain(user_yieldfrom, frame.f_generator.gi_yieldfrom) once (but note that gi_yieldfrom is just for inspection and the interpreter actually uses _PyGen_yf which reads the delegation target from the stack. So we have to overwrite what _PyGen_yf finds. f_generator.gi_yieldfrom uses a getter that just wraps _PyGen_yf).

Maybe just overwrite the stack pointer? We wouldn't need an extra field in every generator-like construct.
Or perhaps monkeypatch the entire f_generator?

Or add a field to generator, say gi_debugging_extra_items and try to yield from it instead of the actual generator in send_ex2. I'll work on that in my mutable-generators branch.

EDIT: This is likely impossible without patching the bytecode to interleave it with dynamic new yields, but I haven't confirmed this.

@gaogaotiantian

@gaogaotiantian Should this be closed or is there more to do?

@kumaraditya303

Hi @kumaraditya303 , the first task is finished but I still want to work on async task list and switch to other async tasks. I will also take a look at the feedbacks of this feature and try to improve it. This will be on the list for my core dev sprint :)

@gaogaotiantian

This will be on the list for my core dev sprint :)

@gaogaotiantian, let's chat about it there!

Navigation Menu

Repository navigation

Make pdb asyncio aware #121468

Feature or enhancement

Proposal:

Has this already been discussed elsewhere?

Links to previous discussion of this feature:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Make pdb asyncio aware #121468

Description

Feature or enhancement

Proposal:

Has this already been discussed elsewhere?

Links to previous discussion of this feature:

Linked PRs

Activity

kumaraditya303 commented on Sep 29, 2024

gaogaotiantian commented on Sep 29, 2024

kumaraditya303 commented on Nov 1, 2024

gaogaotiantian commented on Nov 1, 2024

kumaraditya303 commented on Nov 1, 2024

gaogaotiantian commented on Nov 1, 2024

ktbarrett commented on May 30, 2025

ktbarrett commented on Aug 11, 2025

johnslavik commented on Sep 7, 2025

johnslavik commented on Sep 7, 2025

kumaraditya303 commented on Aug 3, 2026

gaogaotiantian commented on Aug 12, 2026

johnslavik commented on Aug 12, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions