Repository navigation
C-API for signalling monitoring events #111997
Description
Activity
Do you want to propose an API for that?
PEP-669 seems to discuss a mechanism to modify the bytestream for the purpose of implementing monitoring. Would access to this mechanism also be provided as part of this API? I haven't seen too much mention of this feature elsewhere.
Source: https://peps.python.org/pep-0669/#rationale
The quickening mechanism provided by PEP 659 provides a way to dynamically modify executing Python bytecode. These modifications have little cost beyond the parts of the code that are modified and a relatively low cost to those parts that are modified. We can leverage this to provide an efficient mechanism for monitoring that was not possible in 3.10 or earlier.
IMO it would be very useful for an API of this nature to expose low-level interpreter state to monitoring tools. For example, if we are able to see events like
INSTRUCTION, it would also be useful to have access to the operands involved.Do you want to propose an API for that?
In order to map source/line/column information to an offset, we need a way to build a
co_linetablefor a code object: https://github.com/python/cpython/blob/main/Objects/locations.md
That's probably the most complex thing to do here. IIUC, CPython implements this in C here:
Lines 190 to 444 in d67f947
That's purely internal code. I don't think we need C-API support for this, but a Python implementation would be nice. Or a static method on code objects to build the byte string from a Python list of positions. Given the complexity of the format and the risk of future changes, I think it's in CPython's responsibility to generate this format from something user friendly.
compile.hdefines the location like this:/* source location information */ typedef struct { int lineno; int end_lineno; int col_offset; int end_col_offset; } _PyCompilerSrcLocation;
This looks like a function could help that accepts a sequence of tuples of these four values and maps it to a
bytesobject with the corresponding line table in it.We'd probably still have to reimplement it in Cython in order to generate the line table string in "older" Python versions. But we could at least rely on it from CPython 3.13 on.
Reacted by Antoine Pitrou and Mahmoud Rusty Abdelkader@markshannon, could you please comment whether you consider this the right approach? To me, a line table seems required to generate source level monitoring events.
@markshannon any comments? Given that Python 3.12 broke an entire C-API feature, I consider it a blocker for 3.13 to fix it.
What C-API was broken?
AFAICT there was no API to call thetstate->c_tracefuncandtstate->c_profilefuncfunctions.
How were you calling those functions, and why does it no longer work?C extensions can already register callbacks and set events using the Python API, so I assume we are talking about adding an API for C extensions to fire events.
There will be two parts to such an API:
- Specifying the mapping from "code objects" and offsets to full locations.
- Firing events from C extension code.
Handling locations
Apart from
LINEevents, all event callback functions includecode: CodeType, offset: int. Let's not force C extensions to build a full code objects, so we should relaxcodeto be a "code like" object. We define exactly what "code like" means later.We also need a specification for the table mapping offsets to locations. The existing table format is quite fiddly, and can only be searched linearly, so designing a new table format might be a good idea.
Firing events from C extension code
Any potential API has to be more complex than for
sys.settraceas we need to handle disabling of event locations and multiple tools.We will need 16 bits of data for each event location, to track active tools and which tool/location pairs are disabled.
We will need to initialize this data, and to specify which event location maps to which event.
With that in mind, and using the
PY_STARTevent as an example, an API could look something like this:typedef struct _PyMonitoringState { uint8_t active; uint8_t opaque; } PyMonitoringState; typedef struct _PyCodeLikeObject PyCodeLikeObject; /* Returns a version, which should be stored and passed to the next call of _PyMonitioringEnterScope. Arguments: previous_version: Points to a per code-like object value. The value must be set to 0 before first call per code-like object. state_array: The array of all PyMonitoringStates for this code-like object. Should be initialized to all zeroes before first call. event_types: Array of the event types describing the event type for each state in state array. length: The length of the state and event arrays, which must be the same. */ void _PyMonitoringEnterScope(uint64_t *previous_version, PyMonitoringState *state_array, uint8_t event_types, uint32_t length); int _PyMonitoring_FirePyStartEvent(PyCodeLikeObject *codelike, uint32_t offset, PyMonitoringState *state);
If performance is going to be an issue, then the above could be implemented as inline functions.
We would lose ABI compatibility, but would still retain API compatibility across CPython versions.Example usage.
Suppose we have a code-like object that that should fire two events when called,
PY_STARTthenPY_RETURN.We would need to describe the events:
static const uint8_t EVENTS = { PY_START, PY_RETURN };
And the code-like object would need an array for the state and a version number.
struct MyCodeLike { PyObject_HEAD PyMonitoringState monitoring_state[2]; uint64_t version; /* Other fields */ };
The C code for the function:
void init_code(MyCodeLike *code) { code->monitoring_state = {0, 0}; code->version = 0; } PyObject *call_code(MyCodeLike *code) { _PyMonitoringEnterScope(&code->version, code->monitoring_state, EVENTS, 2); ... _PyMonitoring_FirePyStartEvent(code, 0, &code->monitoring_state[0]); ... _PyMonitoring_FirePyReturnEvent(code, 1, &code->monitoring_state[1]); ... }
21 remaining items
We can remove the assert, nothing will crash.
However, there might be an issue with the frame passed to thesys.settracecallback function, as it will be the frame of the closest Python function, not the Cython function.How did Cython work with profiling prior to 3.12? Did you create a fake frame?
How did Cython work with profiling prior to 3.12? Did you create a fake frame?
Yes. We could keep doing that, but a) frames aren't "officially" exposed from the C-API and b) as I understand it, the goal of creating the monitoring C-API was to allow non-Python non-frame code to interact with the tracing infrastructure, so requiring frames seems counter-productive.
I now found that coverage.py was actually updated to work directly with
sys.monitoring, so I tried that and ran into some more issues:File "/home/stefan/source/Python/cython/cython-git/venv/py314/lib/python3.14/site-packages/coverage/sysmon.py", line 337, in sysmon_py_start sys_monitoring.set_local_events( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ self.myid, ^^^^^^^^^^ ...<7 lines>... # | events.JUMP ^^^^^^^^^^^^^^^ ) ^ SystemError: cannot instrument shim code object 'func1'The exception originates here and seems to trigger on generally all code objects that do not have byte code:
cpython/Python/instrumentation.c
Lines 1971 to 1974 in de19694
set_local_events()also has this code, requiring the "code-like object" to be an actual code object:
cpython/Python/instrumentation.c
Lines 2253 to 2259 in de19694
I'm afraid that we might still run into a lot of similar assumptions about what is traceable code.
Changing the code from
if (code->_co_firsttraceable >= Py_SIZE(code)) {
to
if (code->_co_firsttraceable && code->_co_firsttraceable >= Py_SIZE(code)) {
lets me get past the check and continue.- added a commit that references this issue
on May 26, 2024 - added a commit that references this issue
on May 26, 2024 Should we make
sys_trace_line_funcand other functions in legacy_tracing.c create a fake frame if frame passed in is NULL?What are the open issues here currently?
I'm closing this as complete. Let's create new issues for any followup work.
- moved this from In Progress to Done in Release and Deferred blockers 🚫
on Sep 7, 2024
Feature or enhancement
Proposal:
Language implementations for the CPython runtime (Cython, JIT compilers, regular expression engines, template engines, etc.) need a way to signal PEP-669 monitoring events to the registered listeners.
We need a way to create events and inject them into the monitoring system. Since events have more than one signature, we might end up needing more than one C-API function for this.
We need a way to map 3D source code positions (file name, line, character) to 1D integer offsets. Code objects help, but branches might cross source file boundaries, so that's more tricky. For many use cases, a mapping between (line, character) positions and an integer offset would probably suffice, although templating languages usually provide some kind of include commands, as does Cython. There should be some help in the C-API for building up such a mapping.
The reason why I think we need CPython's help for mapping indices is that both sides, listeners and even producers, need to agree on the same mapping. Sadly, the events don't include line/character positions directly but only a single integer. So event producers need a way to produce a number that event listeners like coverage analysers can map back to a source code position.
Has this already been discussed elsewhere?
I have already discussed this feature proposal on Discourse
Links to previous discussion of this feature:
https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018/61?u=scoder
Linked PRs