Add pure Python operator module

@brettcannon

BPO	16694
Nosy	@brettcannon, @rhettinger, @jcea, @pitrou, @ezio-melotti, @merwok, @alex, @bitdancer, @meadori, @zware, @serhiy-storchaka
Files	py_operator.v10.diff: Version 10 py_operator.v11.diff: Version 11, now with proper git format py_operator.v12.diff: Version 12 py_operator.v13.diff: Version 13

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-04-20.17:22:45.334>
created_at = <Date 2012-12-16.07:24:41.590>
labels = ['extension-modules', 'type-feature', 'library']
title = 'Add pure Python operator module'
updated_at = <Date 2013-05-11.02:57:58.758>
user = 'https://github.com/zware'

bugs.python.org fields:

activity = <Date 2013-05-11.02:57:58.758>
actor = 'python-dev'
assignee = 'none'
closed = True
closed_date = <Date 2013-04-20.17:22:45.334>
closer = 'pitrou'
components = ['Extension Modules', 'Library (Lib)']
creation = <Date 2012-12-16.07:24:41.590>
creator = 'zach.ware'
dependencies = []
files = ['28904', '29844', '29869', '29887']
hgrepos = []
issue_num = 16694
keywords = ['patch']
message_count = 39.0
messages = ['177579', '177585', '177587', '177795', '177799', '177810', '177865', '177871', '177895', '177899', '177901', '177902', '177907', '177908', '177910', '177926', '178778', '178787', '178813', '178830', '178838', '180948', '186814', '186883', '186916', '187003', '187007', '187023', '187043', '187046', '187048', '187049', '187050', '187056', '187103', '187148', '187440', '187441', '188890']
nosy_count = 13.0
nosy_names = ['brett.cannon', 'rhettinger', 'jcea', 'pitrou', 'ezio.melotti', 'eric.araujo', 'Arfrever', 'alex', 'r.david.murray', 'meador.inge', 'python-dev', 'zach.ware', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue16694'
versions = ['Python 3.4']

(Brett, I've made you nosy due to the relation to bpo-16651.)

Here is a pure Python implementation of the operator module, or at least a first draft thereof :). I'm attaching the module itself, as well as a patch to integrate it.

Any and all review is quite welcome. I'm confident in the fact that the module as it stands passes all current tests, but how it gets there is entirely up for debate (namely, the attrgetter, itemgetter, and methodcaller classes, as well as length_hint(), countOf(), and indexOf()).

Note that there's also a change to hmac.py; _compare_digest() in operator.c doesn't seem to have any relation to the rest of the module (see bpo-15061 discussion) and is private anyway, so operator.py doesn't go near it. hmac.py has to import directly from _operator.

Thanks,

Zach Ware

Here is a functional (and more effective) equivalent of attrgetter:

def attrgetter(attr, *attrs):
    """
    Return a callable object that fetches the given attribute(s) from its operand.
    After f=attrgetter('name'), the call f(r) returns r.name.
    After g=attrgetter('name', 'date'), the call g(r) returns (r.name, r.date).
    After h=attrgetter('name.first', 'name.last'), the call h(r) returns
    (r.name.first, r.name.last).
    """
    if not attrs:
        if not isinstance(attr, str):
            raise TypeError('attribute name must be a string')
        names = attr.split('.')
        def func(obj):
            for name in names:
                obj = getattr(obj, name)
            return obj
        return func
    else:
        getters = tuple(map(attrgetter, (attr,) + attrs))
        def func(obj):
            return tuple(getter(obj) for getter in getters)
        return func

Perhaps Modules/operator.c should be renamed to Modules/_operator.c.

Also note, that error messages in Python an C implementations sometimes differ.

Sorry to have disappeared on this, other things took priority...

Thank you for the comments, Serhiy. v2 of the patch renames Modules/operator.c to Modules/_operator.c, and changes that name every place I could find it.

I also tried to tidy up some of the error message mismatches. I didn't bother with the ones regarding missing arguments, as that would mean checking args and throwing an exception in each and every function.

I do like the functional attrgetter better than the object version I wrote. The main reason I went with an object version in the first place was because that's what the C implementation used. Is there any reason not to break with the C implementation and use a function instead? The updated patch takes a rather ugly hack to try to use the functional version in an object.

length_hint() was horrible and has been rewritten. It should be less horrible now :). It should also follow the C implementation quite a bit better.

Considering what a huge headache it was to get my own patch to apply at home on Linux rather than at work on Windows, here's a new version of the patch that straightens out the line ending nightmare present in v2. No other changes made.

Sorry, I forgot push a "Publish All My Drafts" button. Please consider other my comments to first patch. I also have added new comments about length_hint().

Your implementation of attrgetter() looks good. One possible disadvantage of pure functional approach is that attrgetter() will be not a class. Unlikely someone subclass attrgetter, but it can be used in an isinstance() check. You solve this issue.

The same approach can be applied to itemgetter().

Here's v4, addressing Serhiy's comments on Reitveld.

About length_hint():

I were mean something like (even explicit getattr() not needed):

try:
hint = type(obj).__length_hint__
except AttributeError:
return default
try:
val = hint(obj)
except TypeError:
return default
...

This is a little faster because there is only one attribute lookup instead two. This is a little safer because there is a little less chance of race when an attribute changed between two lookups (it is enough non-probably and doesn't matter).

There is type(obj) here because the C code uses _PyObject_LookupSpecial() which doesn't honor instance attributes and looks only class attributes.

About concat() and iconcat():

I think only first argument can be checked. If arguments are not concatenable then '+'/'+=' operator will raise an exception. I'm not sure. Does anyone have any thoughts about this?

About methodcaller():

Here is a catch. With this implementation you can't use methodcaller('foo', name='spam') or methodcaller('foo', self='spam') (please add tests for those cases). Here is a trick needed:

def __init__(*args, **kwargs):
    self = args[0]
    self._name = args[1]
    self._args = args[2:]
    self._kwargs = kwargs

(You can add a code for better error reporting).

I have added smaller comments on Rietveld.

Here's another new version. Changes include:

Address Serhiy's Rietveld comments
Fix length_hint() the way it was meant to be fixed last time.
Remove __getitem__ check on 'b' in concat and iconcat. More notes on this below.
Fix methodcaller as Serhiy suggested
Add test case for methodcaller for 'name' and 'self' keyword arguments
Add comments to 'subdivide' the module into the rough sections the docs are divided into. Move length_hint() with other sequence operations to also match the doc order.

On concat and iconcat: Looking at the glossary, a sequence should actually have both __getitem__ and __len__. The test class in the test case for iconcat only defines __getitem__, though. Should we check only for __getitem__ on the first argument, or check for both __getitem__ and __len__, and add __len__ to the test class? Requiring __len__ may cause breakage for anyone using the Python implementation with a class they defined and used with the C implementation with only __getitem__, so I'm leaning towards only checking for __getitem__. I can't really tell what the C implementation really looks for as I don't speak C, but it almost looks to me like it may be only checking for __getitem__. Latest patch only checks argument 'a' for __getitem__.

Good work, Zachary. I have no more nitpicks for you. ;)

LGTM.

One comment to a committer. Don't forget to run hg rename [Modules/operator.c](https://github.com/python/cpython/blob/main/Modules/operator.c) [Modules/_operator.c](https://github.com/python/cpython/blob/main/Modules/_operator.c) before applying the patch.

Nits are no fun; thank you for picking them, Serhiy ;)

I would like to spend some time with this before it goes forward (especially the attrgetter, itemgetter, methodgetter group).

Right now, it looks like a nice effort but I don't see how it makes Python any better for adding it. The odds are that this code will add bloat but not benefit any user (it won't get called at all).

Raymond: it's not for the benefit of CPython.

[David]

Raymond: it's not for the benefit of CPython.

IIRC, all the other implementations of Python already have this code passing tests, so it isn't really for their benefit either.

If a pure python operator module were a part of the stdlib, we (PyPy) would probably delete most (if not all) of our own operator module.

I reviewed the attrgetter(), mathodgetter(), and itemgetter() code in py_operator.v12.diff. The looks clean and correct.

Now we can remove all __func__s from _operator.c.

Thank you for the review, Raymond.

Since Serhiy agrees that the _operator __func__s are unnecessary, here's a v13 that removes them. Again, I'm not a native C speaker, so these new changes in _operator.c deserve a bit of extra scrutiny. Everything builds and still passes the test suite, though.

Also changed in this patch, test_pow and test_inplace remove explicit testing of __func__s. Those tests are useless, as they are merely rerunning already run tests on the same function with a different name, which is confirmed by test_dunder_is_original. I can extend that test with an explicit list of funcs which should have a __func__ if anyone thinks it's worth it.

length_hint() looks ok as well.

New changeset 97834382c6cc by Antoine Pitrou in branch 'default':
Issue bpo-16694: Add a pure Python implementation of the operator module.
http://hg.python.org/cpython/rev/97834382c6cc

I've now commited the latest patch. Thank you very much, Zachary!

New changeset 4b3238923b01 by Raymond Hettinger in branch 'default':
Issue bpo-16694: Add source code link for operator.py
http://hg.python.org/cpython/rev/4b3238923b01

Navigation Menu

Repository navigation

Add pure Python operator module #60898

18 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Add pure Python operator module #60898

Description

Activity

zware commented on Dec 16, 2012

serhiy-storchaka commented on Dec 16, 2012

serhiy-storchaka commented on Dec 16, 2012

zware commented on Dec 20, 2012

zware commented on Dec 20, 2012

serhiy-storchaka commented on Dec 20, 2012

zware commented on Dec 21, 2012

serhiy-storchaka commented on Dec 21, 2012

zware commented on Dec 21, 2012

serhiy-storchaka commented on Dec 21, 2012

serhiy-storchaka commented on Dec 21, 2012

zware commented on Dec 21, 2012

18 remaining items

rhettinger commented on Apr 16, 2013

bitdancer commented on Apr 16, 2013

rhettinger commented on Apr 16, 2013

alex commented on Apr 16, 2013

rhettinger commented on Apr 16, 2013

serhiy-storchaka commented on Apr 16, 2013

zware commented on Apr 16, 2013

pitrou commented on Apr 17, 2013

python-dev commented on Apr 20, 2013

pitrou commented on Apr 20, 2013

python-dev commented on May 11, 2013

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions