Repository navigation
Expand file tree
/
Copy pathgcsupport.rst
More file actions
529 lines (361 loc) · 19.3 KB
/
Copy pathgcsupport.rst
File metadata and controls
529 lines (361 loc) · 19.3 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
.. highlight:: c
.. _supporting-cycle-detection:
Supporting Cyclic Garbage Collection
====================================
Python's support for detecting and collecting garbage which involves circular
references requires support from object types which are "containers" for other
objects which may also be containers. Types which do not store references to
other objects, or which only store references to atomic types (such as numbers
or strings), do not need to provide any explicit support for garbage
collection.
To create a container type, the :c:member:`~PyTypeObject.tp_flags` field of the type object must
include the :c:macro:`Py_TPFLAGS_HAVE_GC` and provide an implementation of the
:c:member:`~PyTypeObject.tp_traverse` handler. If instances of the type are mutable, a
:c:member:`~PyTypeObject.tp_clear` implementation must also be provided.
:c:macro:`Py_TPFLAGS_HAVE_GC`
Objects with a type with this flag set must conform with the rules
documented here. For convenience these objects will be referred to as
container objects.
Constructors for container types must conform to two rules:
#. The memory for the object must be allocated using :c:macro:`PyObject_GC_New`
or :c:macro:`PyObject_GC_NewVar`.
#. Once all the fields which may contain references to other containers are
initialized, it must call :c:func:`PyObject_GC_Track`.
Similarly, the deallocator for the object must conform to a similar pair of
rules:
#. Before fields which refer to other containers are invalidated,
:c:func:`PyObject_GC_UnTrack` must be called.
#. The object's memory must be deallocated using :c:func:`PyObject_GC_Del`.
.. warning::
If a type adds the Py_TPFLAGS_HAVE_GC, then it *must* implement at least
a :c:member:`~PyTypeObject.tp_traverse` handler or explicitly use one
from its subclass or subclasses.
When calling :c:func:`PyType_Ready` or some of the APIs that indirectly
call it like :c:func:`PyType_FromSpecWithBases` or
:c:func:`PyType_FromSpec` the interpreter will automatically populate the
:c:member:`~PyTypeObject.tp_flags`, :c:member:`~PyTypeObject.tp_traverse`
and :c:member:`~PyTypeObject.tp_clear` fields if the type inherits from a
class that implements the garbage collector protocol and the child class
does *not* include the :c:macro:`Py_TPFLAGS_HAVE_GC` flag.
.. c:macro:: PyObject_GC_New(TYPE, typeobj)
Analogous to :c:macro:`PyObject_New` but for container objects with the
:c:macro:`Py_TPFLAGS_HAVE_GC` flag set.
Do not call this directly to allocate memory for an object; call the type's
:c:member:`~PyTypeObject.tp_alloc` slot instead.
When populating a type's :c:member:`~PyTypeObject.tp_alloc` slot,
:c:func:`PyType_GenericAlloc` is preferred over a custom function that
simply calls this macro.
Memory allocated by this macro must be freed with
:c:func:`PyObject_GC_Del` (usually called via the object's
:c:member:`~PyTypeObject.tp_free` slot).
.. seealso::
* :c:func:`PyObject_GC_Del`
* :c:macro:`PyObject_New`
* :c:func:`PyType_GenericAlloc`
* :c:member:`~PyTypeObject.tp_alloc`
.. c:macro:: PyObject_GC_NewVar(TYPE, typeobj, size)
Analogous to :c:macro:`PyObject_NewVar` but for container objects with the
:c:macro:`Py_TPFLAGS_HAVE_GC` flag set.
Do not call this directly to allocate memory for an object; call the type's
:c:member:`~PyTypeObject.tp_alloc` slot instead.
When populating a type's :c:member:`~PyTypeObject.tp_alloc` slot,
:c:func:`PyType_GenericAlloc` is preferred over a custom function that
simply calls this macro.
Memory allocated by this macro must be freed with
:c:func:`PyObject_GC_Del` (usually called via the object's
:c:member:`~PyTypeObject.tp_free` slot).
.. seealso::
* :c:func:`PyObject_GC_Del`
* :c:macro:`PyObject_NewVar`
* :c:func:`PyType_GenericAlloc`
* :c:member:`~PyTypeObject.tp_alloc`
.. c:function:: PyObject* PyUnstable_Object_GC_NewWithExtraData(PyTypeObject *type, size_t extra_size)
Analogous to :c:macro:`PyObject_GC_New` but allocates *extra_size*
bytes at the end of the object (at offset
:c:member:`~PyTypeObject.tp_basicsize`).
The allocated memory is initialized to zeros,
except for the :c:type:`Python object header <PyObject>`.
The extra data will be deallocated with the object, but otherwise it is
not managed by Python.
Memory allocated by this function must be freed with
:c:func:`PyObject_GC_Del` (usually called via the object's
:c:member:`~PyTypeObject.tp_free` slot).
.. warning::
The function is marked as unstable because the final mechanism
for reserving extra data after an instance is not yet decided.
For allocating a variable number of fields, prefer using
:c:type:`PyVarObject` and :c:member:`~PyTypeObject.tp_itemsize`
instead.
.. versionadded:: 3.12
.. c:macro:: PyObject_GC_Resize(TYPE, op, newsize)
Resize an object allocated by :c:macro:`PyObject_NewVar`.
Returns the resized object of type ``TYPE*`` (refers to any C type)
or ``NULL`` on failure.
*op* must be of type :c:expr:`PyVarObject *`
and must not be tracked by the collector yet.
*newsize* must be of type :c:type:`Py_ssize_t`.
.. c:function:: void PyObject_GC_Track(PyObject *op)
Adds the object *op* to the set of container objects tracked by the
collector. The collector can run at unexpected times so objects must be
valid while being tracked. This should be called once all the fields
followed by the :c:member:`~PyTypeObject.tp_traverse` handler become valid, usually near the
end of the constructor.
.. c:function:: int PyObject_IS_GC(PyObject *obj)
Returns non-zero if the object implements the garbage collector protocol,
otherwise returns 0.
The object cannot be tracked by the garbage collector if this function returns 0.
.. c:function:: int PyObject_GC_IsTracked(PyObject *op)
Returns 1 if the object type of *op* implements the GC protocol and *op* is being
currently tracked by the garbage collector and 0 otherwise.
This is analogous to the Python function :func:`gc.is_tracked`.
.. versionadded:: 3.9
.. c:function:: int PyObject_GC_IsFinalized(PyObject *op)
Returns 1 if the object type of *op* implements the GC protocol and *op* has been
already finalized by the garbage collector and 0 otherwise.
This is analogous to the Python function :func:`gc.is_finalized`.
.. versionadded:: 3.9
.. c:function:: void PyObject_GC_Del(void *op)
Releases memory allocated to an object using :c:macro:`PyObject_GC_New` or
:c:macro:`PyObject_GC_NewVar`.
Do not call this directly to free an object's memory; call the type's
:c:member:`~PyTypeObject.tp_free` slot instead.
Do not use this for memory allocated by :c:macro:`PyObject_New`,
:c:macro:`PyObject_NewVar`, or related allocation functions; use
:c:func:`PyObject_Free` instead.
.. seealso::
* :c:func:`PyObject_Free` is the non-GC equivalent of this function.
* :c:macro:`PyObject_GC_New`
* :c:macro:`PyObject_GC_NewVar`
* :c:func:`PyType_GenericAlloc`
* :c:member:`~PyTypeObject.tp_free`
.. c:function:: void PyObject_GC_UnTrack(void *op)
Remove the object *op* from the set of container objects tracked by the
collector. Note that :c:func:`PyObject_GC_Track` can be called again on
this object to add it back to the set of tracked objects. The deallocator
(:c:member:`~PyTypeObject.tp_dealloc` handler) should call this for the object before any of
the fields used by the :c:member:`~PyTypeObject.tp_traverse` handler become invalid.
.. versionchanged:: 3.8
The :c:func:`!_PyObject_GC_TRACK` and :c:func:`!_PyObject_GC_UNTRACK` macros
have been removed from the public C API.
The :c:member:`~PyTypeObject.tp_traverse` handler accepts a function parameter of this type:
.. c:type:: int (*visitproc)(PyObject *object, void *arg)
Type of the visitor function passed to the :c:member:`~PyTypeObject.tp_traverse` handler.
The function should be called with an object to traverse as *object* and
the third parameter to the :c:member:`~PyTypeObject.tp_traverse` handler as *arg*. The
Python core uses several visitor functions to implement cyclic garbage
detection; it's not expected that users will need to write their own
visitor functions.
The :c:member:`~PyTypeObject.tp_clear` handler must be of the :c:type:`inquiry` type, or ``NULL``
if the object is immutable.
.. c:type:: int (*inquiry)(PyObject *self)
Drop references that may have created reference cycles. Immutable objects
do not have to define this method since they can never directly create
reference cycles. Note that the object must still be valid after calling
this method (don't just call :c:func:`Py_DECREF` on a reference). The
collector will call this method if it detects that this object is involved
in a reference cycle.
.. _gc-traversal:
Traversal
---------
The :c:member:`~PyTypeObject.tp_traverse` handler must have the following type:
.. c:type:: int (*traverseproc)(PyObject *self, visitproc visit, void *arg)
Traversal function for a garbage-collected object, used by the garbage
collector to detect reference cycles.
Implementations must call the
*visit* function for each object directly contained by *self*, with the
parameters to *visit* being the contained object and the *arg* value passed
to the handler. The *visit* function must not be called with a ``NULL``
object argument. If *visit* returns a non-zero value, that value should be
returned immediately.
A typical :c:member:`!tp_traverse` function calls the :c:func:`Py_VISIT`
convenience macro on each of the instance's members that are Python
objects that the instance owns.
For example, this is a (slightly outdated) traversal function for
the :py:class:`threading.local` class::
static int
local_traverse(PyObject *op, visitproc visit, void *arg)
{
localobject *self = (localobject *) op;
Py_VISIT(Py_TYPE(self));
Py_VISIT(self->args);
Py_VISIT(self->kw);
Py_VISIT(self->dict);
return 0;
}
.. note::
:c:func:`Py_VISIT` requires the *visit* and *arg* parameters to
:c:func:`!local_traverse` to have these specific names; don't name them just
anything.
Instances of :ref:`heap-allocated types <heap-types>` hold a reference to
their type. Their traversal function must therefore visit the type::
Py_VISIT(Py_TYPE(self));
Alternately, the type may delegate this responsibility by
calling ``tp_traverse`` of a heap-allocated superclass (or another
heap-allocated type, if applicable).
If they do not, the type object may not be garbage-collected.
If the :c:macro:`Py_TPFLAGS_MANAGED_DICT` bit is set in the
:c:member:`~PyTypeObject.tp_flags` field, the traverse function must call
:c:func:`PyObject_VisitManagedDict` like this::
int err = PyObject_VisitManagedDict((PyObject*)self, visit, arg);
if (err) {
return err;
}
Only the members that the instance *owns* (by having
:term:`strong references <strong reference>` to them) must be
visited. For instance, if an object supports weak references via the
:c:member:`~PyTypeObject.tp_weaklist` slot, the pointer supporting
the linked list (what *tp_weaklist* points to) must **not** be
visited as the instance does not directly own the weak references to itself.
The traversal function has a limitation:
.. warning::
The traversal function must not have any side effects. Implementations
may not modify the reference counts of any Python objects nor create or
destroy any Python objects, directly or indirectly.
This means that *most* Python C API functions may not be used, since
they can raise a new exception, return a new reference to a result object,
have internal logic that uses side effects.
Also, unless documented otherwise, functions that happen to not have side
effects may start having them in future versions, without warning.
For a list of safe functions, see a
:ref:`separate section <duringgc-functions>` below.
.. note::
The :c:func:`Py_VISIT` call may be skipped for those members that provably
cannot participate in reference cycles.
In the ``local_traverse`` example above, there is also a ``self->key``
member, but it can only be ``NULL`` or a Python string and therefore
cannot be part of a reference cycle.
On the other hand, even if you know a member can never be part of a cycle,
as a debugging aid you may want to visit it anyway just so the :mod:`gc`
module's :func:`~gc.get_referents` function will include it.
.. note::
The :c:member:`~PyTypeObject.tp_traverse` function can be called from any
thread.
.. impl-detail::
Garbage collection is a "stop-the-world" operation:
even in :term:`free threading` builds, only one thread state is
:term:`attached <attached thread state>` when :c:member:`!tp_traverse`
handlers run.
.. versionchanged:: 3.9
Heap-allocated types are expected to visit ``Py_TYPE(self)`` in
``tp_traverse``. In earlier versions of Python, due to
`bug 40217 <https://bugs.python.org/issue40217>`_, doing this
may lead to crashes in subclasses.
To simplify writing :c:member:`~PyTypeObject.tp_traverse` handlers,
a :c:func:`Py_VISIT` macro is provided.
In order to use this macro, the :c:member:`~PyTypeObject.tp_traverse`
implementation must name its arguments exactly *visit* and *arg*:
.. c:macro:: Py_VISIT(o)
If the :c:expr:`PyObject *` *o* is not ``NULL``, call the *visit*
callback, with arguments *o* and *arg*.
If *visit* returns a non-zero value, then return it.
This corresponds roughly to::
#define Py_VISIT(o) \
if (op) { \
int visit_result = visit(o, arg); \
if (visit_result != 0) { \
return visit_result; \
} \
}
Traversal-safe functions
^^^^^^^^^^^^^^^^^^^^^^^^
The following functions and macros are safe to use in a
:c:member:`~PyTypeObject.tp_traverse` handler:
* the *visit* function passed to ``tp_traverse``
* :c:func:`Py_VISIT`
* :c:func:`Py_SIZE`
* :c:func:`Py_TYPE`: if called from a :c:member:`!tp_traverse` handler,
:c:func:`!Py_TYPE`'s result will be valid for the duration of the handler call
* :c:func:`PyObject_VisitManagedDict`
* :c:func:`PyObject_TypeCheck`, :c:func:`PyType_IsSubtype`,
:c:func:`PyType_HasFeature`
* :samp:`Py{<type>}_Check` and :samp:`Py{<type>}_CheckExact` -- for example,
:c:func:`PyTuple_Check`
* :ref:`duringgc-functions`
.. _duringgc-functions:
"DuringGC" functions
^^^^^^^^^^^^^^^^^^^^
The following functions should *only* be used in a
:c:member:`~PyTypeObject.tp_traverse` handler; calling them in other
contexts may have unintended consequences.
These functions act like their counterparts without the ``_DuringGC`` suffix,
but they are guaranteed to not have side effects, they do not set an exception
on failure, and they return/set :term:`borrowed references <borrowed reference>`
as detailed in the individual documentation.
Note that these functions may fail (return ``NULL`` or ``-1``),
but as they do not set an exception, no error information is available.
In some cases, failure is not distinguishable from a successful ``NULL`` result.
.. c:function:: void *PyObject_GetTypeData_DuringGC(PyObject *o, PyTypeObject *cls)
void *PyObject_GetItemData_DuringGC(PyObject *o)
void *PyType_GetModuleState_DuringGC(PyTypeObject *type)
void *PyModule_GetState_DuringGC(PyObject *module)
int PyModule_GetToken_DuringGC(PyObject *module, void** result)
See :ref:`duringgc-functions` for common information.
.. versionadded:: 3.15
.. seealso::
:c:func:`PyObject_GetTypeData`,
:c:func:`PyObject_GetItemData`,
:c:func:`PyType_GetModuleState`,
:c:func:`PyModule_GetState`,
:c:func:`PyModule_GetToken`,
:c:func:`PyType_GetBaseByToken`
.. c:function:: int PyType_GetBaseByToken_DuringGC(PyTypeObject *type, void *tp_token, PyTypeObject **result)
See :ref:`duringgc-functions` for common information.
Sets *\*result* to a :term:`borrowed reference` rather than a strong one.
The reference is valid for the duration
of the :c:member:`!tp_traverse` handler call.
.. versionadded:: 3.15
.. seealso:: :c:func:`PyType_GetBaseByToken`
.. c:function:: PyObject* PyType_GetModule_DuringGC(PyTypeObject *type)
PyObject* PyType_GetModuleByToken_DuringGC(PyTypeObject *type, const void *mod_token)
See :ref:`duringgc-functions` for common information.
These functions return a :term:`borrowed reference`, which is
valid for the duration of the :c:member:`!tp_traverse` handler call.
.. versionadded:: 3.15
.. seealso::
:c:func:`PyType_GetModule`,
:c:func:`PyType_GetModuleByToken`
Controlling the Garbage Collector State
---------------------------------------
The C-API provides the following functions for controlling
garbage collection runs.
.. c:function:: Py_ssize_t PyGC_Collect(void)
Perform a full garbage collection, if the garbage collector is enabled.
(Note that :func:`gc.collect` runs it unconditionally.)
Returns the number of collected + unreachable objects which cannot
be collected.
If the garbage collector is disabled or already collecting,
returns ``0`` immediately.
Errors during garbage collection are passed to :data:`sys.unraisablehook`.
This function does not raise exceptions.
.. c:function:: int PyGC_Enable(void)
Enable the garbage collector: similar to :func:`gc.enable`.
Returns the previous state, 0 for disabled and 1 for enabled.
.. versionadded:: 3.10
.. c:function:: int PyGC_Disable(void)
Disable the garbage collector: similar to :func:`gc.disable`.
Returns the previous state, 0 for disabled and 1 for enabled.
.. versionadded:: 3.10
.. c:function:: int PyGC_IsEnabled(void)
Query the state of the garbage collector: similar to :func:`gc.isenabled`.
Returns the current state, 0 for disabled and 1 for enabled.
.. versionadded:: 3.10
Querying Garbage Collector State
--------------------------------
The C-API provides the following interface for querying information about
the garbage collector.
.. c:function:: void PyUnstable_GC_VisitObjects(gcvisitobjects_t callback, void *arg)
Run supplied *callback* on all live GC-capable objects. *arg* is passed through to
all invocations of *callback*.
.. warning::
If new objects are (de)allocated by the callback it is undefined if they
will be visited.
Garbage collection is disabled during operation. Explicitly running a collection
in the callback may lead to undefined behaviour e.g. visiting the same objects
multiple times or not at all.
.. versionadded:: 3.12
.. c:type:: int (*gcvisitobjects_t)(PyObject *object, void *arg)
Type of the visitor function to be passed to :c:func:`PyUnstable_GC_VisitObjects`.
*arg* is the same as the *arg* passed to ``PyUnstable_GC_VisitObjects``.
Return ``1`` to continue iteration, return ``0`` to stop iteration. Other return
values are reserved for now so behavior on returning anything else is undefined.
.. versionadded:: 3.12