Feature or enhancement

Proposal:

I had a few proposals to utilize sys.monitoring for pdb and all of them were raising concerns. We had a discussion during language summit and I got feedbacks from several members that breaking backwards compatibility for bdb is not acceptible. I can totally understand the concern so I have a much more conservative proposal to do this. I would like to get opinions from people before start working on it.

TL; DR:

1. All existing code will work exactly as before.
2. sys.monitoring will be an opt-in feature for bdb.
3. Minimum changes are needed for existing debugger (including pdb) to onboard.

Details:

What I propose, is to keep all the code for bdb as it is, and add extra code for sys.monitoring without disturbing the existing methods. Everything will be hidden behind a feature gate so all the existing code will work exactly as before.

In order to use sys.monitoring instead of sys.settrace, the user needs to init bdb.Bdb with an argument (use_monitoring=True?). Then the underlying bdb.Bdb will work in sys.monitoring mode. Ideally, that's the only change the debugger needs to make.

Of course, in reality, if the debugger wants to onboard this feature, it may need a few tweaks. For example, in pdb, debug command toggles sys.settrace to make it possible to debug something while tracing, that needs to be modified. However, the goal is to make it trivial for debugger developers to switch from sys.settrace to sys.monitoring, if they knew how sys.monitoring works.

Let me re-emphasize it in case that's still a confusion - there's nothing the developer needs to do, if they just want the old sys.settrace, everything will simply work because all the new stuff will be behind a feature gate.

If they chose to use sys.monitoring, there will be a few APIs in bdb.Bdb that does not even make sense anymore - trace_dispatch and dispatch_* functions. The documentation already says:

The following methods of Bdb normally don’t need to be overridden.

and a normal debugger should not override those methods anyway (pdb does not). Again, those APIs will still work in sys.settrace mode.

As for pdb, we can also add a feature gate for it. The user can choose between the sys.settrace mode and the sys.monitoring mode. The behaviors in two modes should be identical, except for the performance. We can even make 3.14 a transition version, where the default mechanism is still sys.settrace, and the user can opt-in the sys.monitoring mode by explicitly asking for it (through initialization argument or command line argument). This way, we can get feedbacks from the brave pioneers without disturbing most pdb users. We will have a full year to fix bugs introduced by the mechanism and stablize it.

In 3.15, we can make sys.monitoring the default and still keep sys.settrace as a fallback plan.

So, why bother? Because we can really gain 100x performance with breakpoints. Not only with breakpoints, even without breakpoints, there's a significant overhead to run with debugger attached:

def trace(*args):
    return None

def f():
    start = time.perf_counter()
    fib(22)
    print(time.perf_counter() - start)

f()

The overhead with trace attached is 4x-5x for f() because the call event will still be triggered and even if f_trace == None, the instrumentation is still there and will be executed! We can have an almost zero-overhead debugger and people are very excited about the possibility.

Has this already been discussed elsewhere?

I have already discussed this feature proposal on Discourse

Links to previous discussion of this feature:

#103103
https://discuss.python.org/t/make-pdb-faster-with-pep-669/37674

Linked PRs

• gh-120144: Make it possible to use sys.monitoring for bdb and make it default for pdb #124533
• gh-120144: Disable the CALL event when possible to achieve zero overhead pdb #131390
• gh-120144: Refactor bdb monitoring backend to match settrace behavior #132484