Coding Patterns for Python Extensions

• 1. Introduction
  • 1.1. A Personal Note
  • 1.2. Firstly Why?
    • 1.2.1. Performance
    • 1.2.2. Interface with C/C++ libraries
    • 1.2.3. Less memory
    • 1.2.4. The GIL
  • 1.3. Why Not?
  • 1.4. Alternatives to C Extensions
    • 1.4.1. ctypes
    • 1.4.2. Code Generators
  • 1.5. A Faustian Bargain
  • 1.6. Summary Advice
  • 1.7. Python Documentation
  • 1.8. Project Links
• 2. A Simple Example
  • 2.1. Faster Please
    • 2.1.1. The C Equivalent Function
    • 2.1.2. The Python Interface to C
    • 2.1.3. The Python Module
    • 2.1.4. The setup.py File
    • 2.1.5. Trying it Out
  • 2.2. How Did We Do?
  • 2.3. It’s Not Over Yet
  • 2.4. C Fights Back
  • 2.5. Summary
• 3. PyObjects and Reference Counting
  • 3.1. Access After Free
  • 3.2. Memory Leaks
  • 3.3. Warning on Relying on Reference Counts of Unity or Less
  • 3.4. Python’s Garbage Collector
  • 3.5. Python Terminology
    • 3.5.1. “New” References
    • 3.5.2. “Stolen” References
    • 3.5.3. Warning on “Stolen” References With Containers
    • 3.5.4. “Borrowed” References
  • 3.6. Strong and Weak References
    • 3.6.1. Examples
  • 3.7. A Possible Precautionary Principle
  • 3.8. Summary
• 4. Containers and Reference Counts
  • 4.1. Some Additional Terminology
    • 4.1.1. Discarded References
    • 4.1.2. Abandoned References
  • 4.2. Exploring the CPython C API
  • 4.3. Tuples
    • 4.3.1. PyTuple_SetItem()
    • 4.3.2. PyTuple_SET_ITEM()
    • 4.3.3. Setting and Replacing NULL
    • 4.3.4. PyTuple_Pack()
    • 4.3.5. Py_BuildValue()
    • 4.3.6. Tuple Getters
    • 4.3.7. Summary
  • 4.4. Lists
    • 4.4.1. PyList_SetItem() and PyList_SET_ITEM()
    • 4.4.2. PyList_Append()
    • 4.4.3. PyList_Insert()
    • 4.4.4. Py_BuildValue()
    • 4.4.5. List Getters
    • 4.4.6. Summary
  • 4.5. Dictionaries
    • 4.5.1. PyDict_SetItem()
    • 4.5.2. PyDict_SetDefault()
    • 4.5.3. PyDict_SetDefaultRef() [Python 3.13+]
    • 4.5.4. PyDict_GetItem()
    • 4.5.5. PyDict_GetItemRef() [Python 3.13+]
    • 4.5.6. PyDict_Pop() [Python 3.13+]
    • 4.5.7. Other APIs
  • 4.6. Sets
    • 4.6.1. PySet_Add()
    • 4.6.2. PySet_Discard()
    • 4.6.3. PySet_Pop()
  • 4.7. Summary
    • 4.7.1. Tuples
    • 4.7.2. Lists
    • 4.7.3. Dictionaries
    • 4.7.4. Sets
• 5. Struct Sequence (a namedtuple in C)
  • 5.1. Differences Between a C Struct Sequence and a Python namedtuple
  • 5.2. A Basic C Struct Sequence
    • 5.2.1. Documentation String
    • 5.2.2. Field Specifications
    • 5.2.3. Struct Sequence Type Specification
    • 5.2.4. Creating an Instance
    • 5.2.5. Using an Instance
  • 5.3. Whether to Provide Access to the Type from Python
    • 5.3.1. Exposing the Type from the CPython Module
    • 5.3.2. Hiding the Type in the Module
  • 5.4. Converting a C struct to a namedtuple
  • 5.5. Controlling Member Access
    • 5.5.1. The Importance of the n_in_sequence Field
  • 5.6. Unnamed Fields
    • 5.6.1. Field Declaration
    • 5.6.2. Type Declaration
    • 5.6.3. Struct Sequence Creation
    • 5.6.4. Access from Python
• 6. Exception Raising
  • 6.1. Common Exception Patterns
    • 6.1.1. Type Checking
  • 6.2. Creating Specialised Exceptions
    • 6.2.1. Declaring Specialised Exceptions
    • 6.2.2. Example Module
    • 6.2.3. Initialising Specialised Exceptions
    • 6.2.4. Raising Specialise Exceptions
• 7. A Pythonic Coding Pattern for C Functions
  • 7.1. Principle
  • 7.2. Coding the Function
  • 7.3. The Function Code as One
• 8. Parsing Python Arguments
  • 8.1. Specifying the Function Arguments
    • 8.1.1. C Function Declaration
• 9. Creating New Types
  • 9.1. Properties
    • 9.1.1. Referencing Existing Properties
    • 9.1.2. Created Properties
  • 9.2. Subclassing
  • 9.3. Examples
    • 9.3.1. Setting, Getting and Deleting Attributes Dynamically
  • 9.4. Emulating Sequence Types
    • 9.4.1. The Sequence Function Table
    • 9.4.2. sq_length
    • 9.4.3. sq_concat
    • 9.4.4. sq_repeat
    • 9.4.5. sq_item
    • 9.4.6. sq_ass_item
    • 9.4.7. sq_ass_contains
    • 9.4.8. sq_inplace_concat
    • 9.4.9. sq_inplace_repeat
    • 9.4.10. The Function Table
    • 9.4.11. Module Initialisation
  • 9.5. TODOs:
• 10. Setting and Getting Module Globals
  • 10.1. Initialising Module Globals
  • 10.2. Getting and Setting
    • 10.2.1. From Python
    • 10.2.2. Getting Module Globals From C
    • 10.2.3. Setting Module Globals From C
• 11. Logging and Frames
  • 11.1. Logging From C
    • 11.1.1. Logging C Declarations
    • 11.1.2. Logging C Globals
    • 11.1.3. Logging C Functions
    • 11.1.4. cLogging Module
    • 11.1.5. Using and Testing
  • 11.2. Frames
    • 11.2.1. The Python Frame
    • 11.2.2. The C “Frame”
• 12. File Paths and Files
  • 12.1. File Paths
    • 12.1.1. Parsing File Paths as Arguments
  • 12.2. Files
    • 12.2.1. Reading a Python File
    • 12.2.2. Writing to a Python File
• 13. Subclassing and Using super()
  • 13.1. Basic Subclassing
    • 13.1.1. Writing the C Extension
    • 13.1.2. Setup and Build
    • 13.1.3. Test
  • 13.2. Calling super() from C
    • 13.2.1. The Obvious Way is Wrong
    • 13.2.2. Doing it Right
  • 13.3. Subclassing with Overloading
    • 13.3.1. Writing the C Extension
    • 13.3.2. Setup and Build
    • 13.3.3. Test
    • 13.3.4. Another Example of Using this API
• 14. Capsules
  • 14.1. A Simple Example
    • 14.1.1. Basic Extension
    • 14.1.2. An Exportable Extension
    • 14.1.3. A Client Extension
  • 14.2. Using an Existing Capsule
    • 14.2.1. Writing the Code for the Object
    • 14.2.2. Building
    • 14.2.3. Testing
• 15. Iterators and Generators
  • 15.1. Iterators
    • 15.1.1. Example of a Sequence
    • 15.1.2. Adding an Iterator
    • 15.1.3. A Note on Module Initialisation
    • 15.1.4. Iterating a Python Object in C
    • 15.1.5. Examples
    • 15.1.6. Reverse Iterators
  • 15.2. Generators
    • 15.2.1. Our Iterator as a Generator
• 16. Context Managers
  • 16.1. C Functions
    • 16.1.1. __enter__
    • 16.1.2. __exit__
    • 16.1.3. Method declarations
  • 16.2. Understanding the Context Manager
    • 16.2.1. A Context Manager Without a Target
    • 16.2.2. A Context Manager With a Target
  • 16.3. Minimal Context Manager in C
    • 16.3.1. __enter__ and __exit__ Methods
    • 16.3.2. Type Declaration
    • 16.3.3. Module
    • 16.3.4. Setup
    • 16.3.5. Testing
• 17. Pickling C Extension Types
  • 17.1. Pickle Version Control
  • 17.2. Implementing __getstate__
  • 17.3. Implementing __setstate__
    • 17.3.1. Error Checking
    • 17.3.2. Set the first Member
    • 17.3.3. Set the last Member
    • 17.3.4. Set the number Member
    • 17.3.5. __setstate__ in Full
  • 17.4. Add the Special Methods
  • 17.5. Pickling a custom2.Custom Object
  • 17.6. The Pickled Object in Detail
  • 17.7. Pickling Objects with External State
  • 17.8. References
• 18. Watchers [Python 3.12+]
  • 18.1. Dictionary Watchers
    • 18.1.1. Low Level C Implementation
    • 18.1.2. Exposing This to CPython
    • 18.1.3. Creating the Context Manager
    • 18.1.4. Module, Setup and Test
    • 18.1.5. Without the Context Manager
  • 18.2. Type Watchers
  • 18.3. Function Watchers
  • 18.4. Code Watchers
• 19. Setting Compiler Flags
  • 19.1. From the Command Line
  • 19.2. Programmatically from Within a Python Process
  • 19.3. From the Command Line using sysconfig
  • 19.4. Setting Flags Automatically in setup.py
  • 19.5. Getting C/C++ Flags from a CMake Build
• 20. Debugging
  • 20.1. Debugging Tools
    • 20.1.1. Build a Debug Version of Python
    • 20.1.2. Valgrind
    • 20.1.3. A Simple Memory Monitor
  • 20.2. Building and Using a Debug Version of Python
    • 20.2.1. Building a Standard Debug Version of Python
    • 20.2.2. Specifying Macros
    • 20.2.3. The Debug Builds
    • 20.2.4. Python’s Memory Allocator
    • 20.2.5. Identifying the Python Build Configuration from the Runtime
  • 20.3. Valgrind
    • 20.3.1. Building Valgrind
    • 20.3.2. Building Python for Valgrind
    • 20.3.3. Using Valgrind
  • 20.4. Leaked New References
    • 20.4.1. A Leak Example
    • 20.4.2. Recognising Leaked New References
    • 20.4.3. Finding Where the Leak is With Valgrind
  • 20.5. Debugging Tactics
    • 20.5.1. Access After Free
  • 20.6. Using gcov for C/C++ Code Coverage
    • 20.6.1. gcov under Mac OS X
    • 20.6.2. Using gcov on CPython Extensions
  • 20.7. Debuging Python C Extensions in an IDE
    • 20.7.1. Creating a Python Unit Test to Execute
    • 20.7.2. Writing a C Function to call any Python Unit Test
    • 20.7.3. Debugging Python C Extensions in Xcode
    • 20.7.4. Using a Debug Version of Python C with Xcode
    • 20.7.5. Debugging Python C Extensions in Eclipse
  • 20.8. Instrumenting the Python Process for Your Structures
    • 20.8.1. An Implementation of a Counter
    • 20.8.2. Instrumenting the Counter
• 21. Memory Leaks
  • 21.1. Introduction
    • 21.1.1. Is it a Leak?
    • 21.1.2. Sources of Leaks
    • 21.1.3. A Bit About (C)Python Memory Management
  • 21.2. Tools for Detecting Memory Leaks
    • 21.2.1. Platform Tools
    • 21.2.2. Python Tools
    • 21.2.3. Debugging Tools
    • 21.2.4. Building a Debug Version of Python
  • 21.3. Techniques
    • 21.3.1. High Level
  • 21.4. pymemtrace
    • 21.4.1. pymemtrace Tools
    • 21.4.2. Tool Characteristics
    • 21.4.3. pymemtrace Process
    • 21.4.4. pymemtrace cPyMemTrace
    • 21.4.5. pymemtrace and DTrace
    • 21.4.6. pymemtrace trace_malloc
    • 21.4.7. pymemtrace Debug Malloc Stats
• 22. Thread Safety
  • 22.1. When You Need a Lock
  • 22.2. Coding up the Lock
    • 22.2.1. Adding a PyThread_type_lock to our object
    • 22.2.2. Initialising and Deallocating the Lock
  • 22.3. Using the Lock
    • 22.3.1. From C Code
    • 22.3.2. From C++ Code
  • 22.4. Example Code and Tests
    • 22.4.1. Example Code
    • 22.4.2. Example Tests
• 23. Source Code Layout
  • 23.1. Testing CPython Utility Code
• 24. Using C++ With CPython Code
  • 24.1. C++ RAII Wrappers Around PyObject*
    • 24.1.1. C++ RAII Wrapper for a Borrowed PyObject*
    • 24.1.2. C++ RAII Wrapper for a New PyObject*
  • 24.2. Handling Default Arguments
    • 24.2.1. Immutable Default Arguments
    • 24.2.2. Mutable Default Arguments
  • 24.3. Homogeneous Python Containers and C++
    • 24.3.1. A Problematic Example
  • 24.4. A CPython Extension containing C++ Objects
    • 24.4.1. Allocation of C++ Objects and Placement new
  • 24.5. Python Unicode Strings and C++
    • 24.5.1. Basic Handling of Unicode
    • 24.5.2. Working with bytes, bytearray and UTF-8 Unicode Arguments
  • 24.6. C++ and the Numpy C API
    • 24.6.1. Initialising Numpy
    • 24.6.2. Verifying Numpy is Initialised
    • 24.6.3. Numpy Initialisation Techniques
  • 24.7. C++ and the Python Buffer Protocol
    • 24.7.1. References:
• 25. Miscellaneous
  • 25.1. No PyInit_... Function Found
  • 25.2. Migrating from Python to a C Extension
• 26. Installation
  • 26.1. Project Links
  • 26.2. Setup
  • 26.3. Stable release
  • 26.4. From sources
  • 26.5. Install Requirements
  • 26.6. What is in the Package
  • 26.7. Running the Tests
  • 26.8. Building the Documentation
  • 26.9. Building Everything
• 27. Further Reading
  • 27.1. C Extensions - General
  • 27.2. Books
  • 27.3. Projects
• 28. TODO
  • 28.1. Existing TODOs
  • 28.2. Work Estimate for v0.3.0 Release
    • 28.2.1. New Chapter “Debugging Python with CLion”
    • 28.2.2. TOTAL
• 29. History
  • 29.1. 0.3.0 (2025-03-20)
    • 29.1.1. Added Chapters
    • 29.1.2. Changed Chapters
    • 29.1.3. Other
    • 29.1.4. TODO
  • 29.2. 0.2.2 (2024-10-21)
  • 29.3. 0.2.1 (2024-07-29)
    • 29.3.1. Contributors
  • 29.4. 0.1.0 (2014-09-09)

Search

• Index

• Module Index

• Search Page