Repository navigation
access to cdecimal / libmpdec API #66390
Description
| BPO | 22194 |
|---|---|
| Nosy | @malemburg, @loewis, @abalkin, @pitrou, @scoder, @larryhastings, @skrah, @pganssle |
| Superseder | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2020-07-17.11:24:39.988>
created_at = <Date 2014-08-14.13:29:57.981>
labels = ['type-feature']
title = 'access to cdecimal / libmpdec API'
updated_at = <Date 2020-07-17.11:24:39.987>
user = 'https://github.com/pitrou'bugs.python.org fields:
activity = <Date 2020-07-17.11:24:39.987>
actor = 'skrah'
assignee = 'none'
closed = True
closed_date = <Date 2020-07-17.11:24:39.988>
closer = 'skrah'
components = []
creation = <Date 2014-08-14.13:29:57.981>
creator = 'pitrou'
dependencies = []
files = ['36378']
hgrepos = []
issue_num = 22194
keywords = ['patch']
message_count = 31.0
messages = ['225297', '225357', '225588', '225589', '225590', '225599', '225762', '225768', '225805', '225821', '225825', '225826', '225827', '225828', '226035', '226176', '226299', '226304', '226310', '226311', '226313', '226314', '226317', '226318', '226322', '226323', '228427', '228452', '228453', '228474', '373825']
nosy_count = 8.0
nosy_names = ['lemburg', 'loewis', 'belopolsky', 'pitrou', 'scoder', 'larry', 'skrah', 'p-ganssle']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '41324'
type = 'enhancement'
url = 'https://bugs.python.org/issue22194'
versions = []
Activity
pitrou commentedon Aug 14, 2014
Currently cdecimal exports no C API that I know of, and it makes sure the libmpdec symbols are kept private in the .so file. It would be nice for external C code (or, in general, non-Python code) to be able to access cdecimal objects, and make operations on them, without the huge overhead of the regular C Python API.
skrah commentedon Aug 15, 2014
I'm a little unsure what to do with the API, see also bpo-15237:
I'm not too fond of the Capsule method, especially because
it *seems* possible to get at the symbols directly on Linux
and Windows (provided that they aren't static of course).
I would not like to spend time on it if we go ahead and
make decimal a builtin (double effort).
It's not clear whether users would not be served better by
using functions from libmpdec directly (much faster,
probably less error handling).
For 3) I've attached a draft that illustrates the issue.
pitrou commentedon Aug 20, 2014
That's what I meant. The issue here is that Python's libmpdec is not exposed to third-party code at all. Also there should probably be a (thin?) API to get at the underlying mpdec object from a cdecimal PyObject (apologies for the poor wording, I'm actually not acquainted with the libmpdec APIs).
As for the Capsule method, well, at least it would be better than nothing (or than any platform-specific hack).
I haven't heard any consensus on that yet :-)
(for the record, the context is that we would like to support decimal objects efficiently in Numba)
pitrou commentedon Aug 20, 2014
Note that you could expose a C API even if decimal didn't become built-in, see Include/datetime.h for example.
pitrou commentedon Aug 20, 2014
Relatedly, is the libmpdec ABI stable? That is, if I build a separate libmpdec, can I expect it to handle cdecimal's innards fine, or will there be problems?
scoder commentedon Aug 21, 2014
Same for Cython, although I guess we wouldn't do more than shipping the necessary declarations and (likely) also enable auto-coercion between the libmpdec decimal type (struct?) and CPython's decimal type, in the same way that we do it for byte strings.
Thus, a public header file with the necessary type checking and packing/unpacking C-API functions would be nice and also sufficient for now.
skrah commentedon Aug 23, 2014
People were asking for libmpdec symbols to be hidden (bpo-16745). It's easy to
revert, just a couple of pragmas in the headers.
Platform specific maybe, but no hack: I was thinking about storing the DSO
handle in the PyModuleObject struct and add functions to lookup symbols.
I'm attaching a diff for Linux -- It has been a while, but I'm rather certain
that the corresponding scheme also worked on Windows (can't test now, my
Windows VM is defunct).
That would leave the usual troublemakers AIX etc., which have sketchy support
anyway.
Of course getting all symbols for a module should be done in one separate
file (_decimal_api.c), the diff is just a demonstration.
Yes, starting from 2.4.0.
You need to initialize libmpdec in the exact same way as in _decimal.c.
Generally it should be fine (though I'm not thrilled by the idea :), but
I suspect there could be locale problems on Windows with a second libmpdec.
pitrou commentedon Aug 23, 2014
Le 23/08/2014 16:42, Stefan Krah a écrit :
Who are those people? bpo-16745 was opened by you :-)
How does it work? I've tried to dlopen() and then dlsym() the _decimal
file manually, it wouldn't work for private (e.g. mpd) symbols.
That sounds rather worrying. How about OS X? Why would that whole scheme
be better than a capsule?
skrah commentedon Aug 24, 2014
Antoine Pitrou <report@bugs.python.org> wrote:
MvL, in bpo-4555 (msg176486).
Yes, the symbols would need to be public, see module_get_symbol.diff.
I'm not sure about OS X, but I would be surprised if it did not work.
To my limited knowledge, Capsules are slow, see also here (the penultimate
paragraph):
https://mail.python.org/pipermail/python-dev/2013-March/124481.html
Stefan (Behnel), could you comment on the strategy that you had in mind?
Is it similar to module_get_symbol.diff or entirely different?
pitrou commentedon Aug 24, 2014
Le 24/08/2014 04:56, Stefan Krah a écrit :
They are slow if you have to lookup and unwrap a capsule every time you
use it. But the way it would work for _decimal (and the way it already
works for _datetime), AFAIK, would be different: you would look up the
"capsule API structure" once and then simply dereference function
pointers from that structure.
You can actually take a look at Include/datetime.h and see if the
approach would work. See especially:
/* Define global variable for the C API and a macro for setting it. */
static PyDateTime_CAPI *PyDateTimeAPI = NULL;
which encourages a once-in-a-process-lifetime API lookup pattern.
I don't think a hand-coded dlsym()-based approach can be any
significantly faster.
skrah commentedon Aug 24, 2014
Ah yes, the array of function pointers is directly accessible. I did not look
close enough -- reading the word "spam" 100x in the docs always makes me skim
the text. ;)
pitrou commentedon Aug 24, 2014
Ok, I'm cc'ing Martin then :-)
Note RTLD_LOCAL seems to be the default with dlopen(). Now I don't know how that behaves when you have a chained library loading, e.g.:
Apache --> dlopen(Python dll) --> dlopen(_decimal dll)
_decimal is an interesting case, since AFAIK with other C extensions it isn't really interesting to access the underlying C library, but with _decimal not being able to access libmpdec would force _decimal to duplicate a large part of the libmpdec API with a PyDec prefix added in front of it.
(which means that, perhaps, the answer is to make the mpd_ prefix configurable with a #define?)
skrah commentedon Aug 24, 2014
Antoine Pitrou <report@bugs.python.org> wrote:
I don't know 100% what you have in mind, but Debian and Arch already ship
--with-system-libmpdec, so only the mpd_* functions would be available.
pitrou commentedon Aug 24, 2014
Le 24/08/2014 11:11, Stefan Krah a écrit :
Ah... that probably kills the idea then. I was thinking in the context
of a private-built libmpdec and had forgotten about that possibility.
4 remaining items
pitrou commentedon Sep 3, 2014
Ah, so it has an incompatible ABI? That will complicate things a bit :-)
pitrou commentedon Sep 3, 2014
Does it mean a separate Windows and AIX solution should be found?
I think if we can't make the mpd symbols available in a cross-platform way, the incentive starts getting much lower :-)
skrah commentedon Sep 3, 2014
Antoine Pitrou <report@bugs.python.org> wrote:
Yes, cdecimal on PyPI is slower, has a different ABI, uses libmpdec-2.3,
has subtle differences in the context handling, cannot subclass the
context, ... ;)
I think a common C API for 2.7 is only possible with a real backport.
pitrou commentedon Sep 3, 2014
Le 03/09/2014 18:33, Stefan Krah a écrit :
Ok, so let's ignore that and focus on 3.5?
skrah commentedon Sep 3, 2014
Sure, if there are people who write python3-only C modules (I can't think
of one right now).
pitrou commentedon Sep 3, 2014
If there aren't today, there will be in a few years time, and they'll be glad they can support Python 3.5.
larryhastings commentedon Sep 3, 2014
Are there any other modules where the capsule API works in both CPython and PyPy? I thought capsule APIs were decidedly implementation-specific.
Not that I'm not for it in theory. But this is some crazy uncharted hyper-compatibility territory we're talking about here.
skrah commentedon Sep 3, 2014
The compatibility discussion was for the cdecimal-2.3 package that's
hosted on PyPI and used for Python 2.7. IOW, will people use a capsule
API that's Python-3 only?
Compatibility with pypy would be esoteric indeed. :)
loewis commentedon Oct 4, 2014
I'd like to focus this issue a bit. Antoine originally proposed that non-Python code might want to access libmpdec. However, given that this is now a separate project (as it seems), I don't think it's Python's task to make the API available. If it is a separate library really, you shouldn't need to link to Python in order to use it. IOW, the solution should be to use the system libmpdec.
So the focus should be on other Python C modules that want to access decimal objects. Are there precedents of such modules? What API features do they actually need?
skrah commentedon Oct 4, 2014
libmpdec has always been a separate project, predating the integration
into Python-3.3. Before the Python-3.3 release, Jim Jewett suggested
a cleaner library/module separation (and he was right, it made the
code much more readable).
Then distributors wanted --with-system-libmpdec, so here we are.
Let's discuss the _decimal capsule API in bpo-15237.
As for the libmpdec symbols: I agree it is somewhat unclean to
do this on the Python side.
For *some* functions it is also tricky (but by no means impossible)
to use them directly and expect the same results as from _decimal.
Maybe a solution for numba is to just assume --with-system-libmpdec.
libmpdec is extremely easy to package, distributions just have to
ensure that their Python version matches the libmpdec version.
skrah commentedon Oct 4, 2014
Of course Windows is a problem. I do not know how to implement
--with-system-libmpdec on Windows.
pitrou commentedon Oct 4, 2014
For the record, we have realized that pure C code brought less than a 2x speedup compared to straightforward Python code, so cdecimal support has become a low-priority concern for Numba. For the curious, the C code is from libmpdec's own benchmark (Web site seems down at the moment) and the Python code is here: https://gist.github.com/pitrou/9eb2a5651b99f2e3c4ce
skrah commentedon Jul 17, 2020
Closing in favor of bpo-41324, which adds just the most important
functions.