C API for simplifying module attribute declaration

Some years ago, Christian created an issue, and a proposed implementation, for C API that could reduce the amount of boiler-plate code needed when implementing extension modules. Quoting the issue description:

It's currently inconvenient and sometimes error-prone to initialize module members in C. Even stdlib modules are sometimes missing error checks or have reference counting issues in error path. I propose add three helpers to deal with common cases:

declaration of simple attributes (int, str, bool, float) in an array of structs

creation and addition of a new type from a type spec

creation and addition a new exception object

The issue and PR received some initial interest (and I remember being positive to adding these helpers), but the discussion came to a halt after Christian took a break from CPython. Can the C API WG help decide if we should add such APIs? If you suggest these APIs are worth it to implement and maintain, I can volunteer to pick up Christan's work update the PR. If you suggest this is not worth it, we can close the issue and the PR.

Hello,
This is sitting in my TODO list, but I wanted to tackle it in a roundabout way:

rewrite the tutorial to use modern API
look at the shortcomings that uncovers
design better “shortcut” API based on that. (e.g. if it doesn't come up in the tutorial, it's likely to be fine as two API calls rather than one.)

AFAIK, Christian's less controversial proposals were merged, and the remaining PR has the ones with potential issues.

PyModuleConst_Def adds a “tagged union”, which might be better designed together with other new “slot” API we'll need
the PyType_FromModuleAndSpec family of functions grows a new member every once in a while; each time that happens we'd need to add a PyModule_Add* variant for it.
Let's review PyErr_NewExceptionWithDoc before adding PyModule_AddNewException.
Could PyModule_AddType work better for the PyModule_AddNewTypeFromSpec and PyModule_AddNewException use cases? If we let it return NULL if it gets type=NULL, so users can skip one type check between creating a type and and adding it.

Extract of proposed PyModule_AddConstants() API:

enum _PyModuleConst_type {
    _PyModuleConst_none_type = 1,
    _PyModuleConst_long_type = 2,
    _PyModuleConst_ulong_type = 3,
    _PyModuleConst_bool_type = 4,
    _PyModuleConst_double_type = 5,
    _PyModuleConst_string_type = 6,
    _PyModuleConst_call_type = 7,
};

typedef struct PyModuleConst_Def {
    const char *name;
    enum _PyModuleConst_type type;
    union {
        const char *m_str;
        long m_long;
        unsigned long m_ulong;
        double m_double;
        PyObject* (*m_call)(PyObject *module);
    } value;
} PyModuleConst_Def;

PyAPI_FUNC(int) PyModule_AddConstants(PyObject *, PyModuleConst_Def *);

#define PyModuleConst_None(name) \
    {(name), _PyModuleConst_none_type, {.m_long=0}}
#define PyModuleConst_Long(name, value) \
    {(name), _PyModuleConst_long_type, {.m_long=(value)}}

I have some concerns:

Macros are not available in Rust. I'm not sure if creating PyModuleConst_Def instance in other programming languages is easy. _PyModuleConst_type values are not part of the API.
PyModuleConst_Def.type uses an enum, whereas enum is not great for the stable ABI.
PyModuleConst_Def.value uses an union. Is it ok with other programming languages such as Rust for a stable ABI?

Maybe it's ok to not expose such "convenient API" in the stable ABI and make it usable for other programming languages, since it's only a layer on top of the existing API such as PyModule_AddIntConstant().

PyAPI_FUNC(PyTypeObject *) PyModule_AddNewTypeFromSpec(PyObject *module, PyType_Spec *spec, PyObject *base);

Python has many functions to create a type from a spec. I would prefer to not duplicate them in the PyModule_Add API family. Can't we add types in PyModule_AddConstants() API instead?

Overall, I like the "PyModule_AddConstants" API. It's just difficult to make it generic and support the stable ABI and most programming languages.

For the simple cases, maybe we can pass an array of PyModuleConst_Def as a "slot" in PyModuleDef.m_slots, to even avoid having to call explicitly PyModule_AddConstants().

So PyModule_AddConstants is just a convenience, right? You could just as well write a bunch of code that creates the objects and adds them to the module one by one -- it's just a bit of boilerplate.

So on the one hand that's a reason why the "but what about Rust" argument doesn't bother me too much -- Rust code can do what it needs to do, and they could even define their own type-safe macros.

OTOH, I'm not convinced that this is a big enough issue to warrant the complexity of yet another convenience API. I'd say that modules with lots of simple global constants are probably more a code smell than a pattern we want to encourage? At the very least it should be just as easy to add a list of such constants to another namespace (e.g. a class or perhaps an enum).

FTR, I'm not interested in following up the issue or PR. Currently we use C macros for simplifying extension module initialization; I'm fine with that. ISTM there is little interest in driving this forward, so I would suggest to close the python/cpython issue/PR as not-planned. If/when Christan comes back to CPython and wishes to follow this up, he can easily reopen the issue and rebase his branch.

@gvanrossum

Not just Christian. It might well be me. But I agree that there's no reason to keep the issue on CPython's tracker.

Proposal: close the PR and issue.

Since this is easy to revert and I don't see anyone opposed, I'll go ahead and close them. We can reopen if anyone disagrees.

We can close this issue after everyone votes.

The current API to initialize a module is really annoying to use, since each call must be checked for error. I'm interested by anything which make it easier.

Currently, since it's annoying to check for errors, many stdlib C extensions simply don't check for errors. That's bad. Others use complex macros to inject error handling ("return" or "goto" on error), without making the code too verbose to use.

The current API to initialize a module is really annoying to use, since each call must be checked for error.

That goes for pretty much any API; most APIs can fail, so you have to add error handling.

@erlend-aasland

@erlend-aasland:

FTR, I'm not interested in following up the issue or PR.

Oh, I first understood that you was interested to pursue the effort. Sadly, without any sponsor, this issue is not going anywhere. Ok to close it.

See you all again :). I ended up coming back to this issue while pushing python/cpython#127435 and I think it's OK to reopen it and I'll follow up with it, but we need to work out a final solution.

Based on the above discussion, I make the following conclusions:

Reuse existing PyModule_* functions instead of adding new APIs.
Add a new structure and corresponding macro to describe this process.

So, what do you think of this incident?

@rruuaanng

Sorry, @rruuaanng, I don't understand what you are proposing.

@rruuaanng

Sorry, @rruuaanng, I don't understand what you are proposing.

@encukou Please allow us to discuss this privately :)

Navigation Menu

Repository navigation

C API for simplifying module attribute declaration #6

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

C API for simplifying module attribute declaration #6

Description

Activity

encukou commented on Jan 5, 2024

vstinner commented on Jan 10, 2024

vstinner commented on Jan 10, 2024

gvanrossum commented on Jan 10, 2024

erlend-aasland commented on Jan 11, 2024

encukou commented on Jan 11, 2024

vstinner commented on Jan 11, 2024

erlend-aasland commented on Jan 11, 2024

vstinner commented on Jan 15, 2024

rruuaanng commented on Nov 30, 2024

encukou commented on Dec 4, 2024

rruuaanng commented on Dec 4, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions