Add AutoNumberedEnum to stdlib

@warsaw

BPO	26988
Nosy	@warsaw, @rhettinger, @vstinner, @ethanfurman, @vedgar, @Vgr255, @johnthagen, @kennethreitz
Files	issue26988.stoneleaf.01.patch issue26988.stoneleaf.02.patch issue26988.stoneleaf.03.patch issue26988.stoneleaf.05.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ethanfurman'
closed_at = <Date 2016-08-20.07:02:02.617>
created_at = <Date 2016-05-10.01:34:31.690>
labels = ['type-feature', 'library']
title = 'Add AutoNumberedEnum to stdlib'
updated_at = <Date 2016-08-20.07:02:02.616>
user = 'https://github.com/johnthagen'

bugs.python.org fields:

activity = <Date 2016-08-20.07:02:02.616>
actor = 'ethan.furman'
assignee = 'ethan.furman'
closed = True
closed_date = <Date 2016-08-20.07:02:02.617>
closer = 'ethan.furman'
components = ['Library (Lib)']
creation = <Date 2016-05-10.01:34:31.690>
creator = 'John Hagen'
dependencies = []
files = ['43694', '43743', '43972', '43986']
hgrepos = []
issue_num = 26988
keywords = ['patch']
message_count = 62.0
messages = ['265214', '265215', '265216', '265218', '265237', '269476', '269521', '270039', '270048', '270049', '270051', '270054', '270056', '270071', '270088', '270107', '270117', '270137', '270142', '270148', '270151', '270199', '270201', '270212', '270219', '270238', '270523', '270524', '271811', '271855', '271868', '272071', '272874', '272876', '272885', '272901', '272932', '272941', '272951', '272952', '272971', '272982', '273006', '273039', '273040', '273064', '273065', '273076', '273084', '273092', '273103', '273105', '273109', '273118', '273125', '273126', '273127', '273131', '273134', '273145', '273154', '273188']
nosy_count = 11.0
nosy_names = ['barry', 'rhettinger', 'vstinner', 'eli.bendersky', 'ethan.furman', 'python-dev', 'veky', 'abarry', 'John Hagen', 'David Hagen', 'kennethreitz']
pr_nums = []
priority = 'normal'
resolution = 'rejected'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue26988'
versions = ['Python 3.6']

I suggest that the AutoNumberedEnum be added to the standard library for the following reasons:

Provides a fundamental tool for defining a unique, abstract set of coupled members
Avoids boilerplate @enum.unique for a very common use case of enumerations
The code already exists in the Python documentation, so it has been vetted at some level

The AutoNumberedEnum also allows the developer to make a clearer distinction
between enumerations whose values have special meaning and those that do not.

Consider:

@enum.unique
class Color(enum.Enum):
    red = 1
    blue = 2
    green = 3

@enum.unique
class Shape(enum.Enum):
    """Member values denote number of sides."""
    circle = 1
    triangle = 3
    square = 4

With AutoNumberedEnum it's possible to better express the intent that
the value of Color members does not hold special meaning, while
Shape members do:

class Color(enum.AutoNumberedEnum):
    red = ()
    blue = ()
    green = ()

@enum.unique
class Shape(enum.Enum):
    """Member values denote number of sides."""
    circle = 1
    triangle = 3
    square = 4

For enumerations with many members (10s), there becomes a maintenance
issue when inserting new enumerations into the list:

@enum.unique
class Color(enum.Enum):
    aquamarine = 1
    blue = 2
    fushia = 3
    # inserting a member here (perhaps because it's clearest to keep these in alphabetic order)
    # results in having to increment all following members
    ...
    green = 40
    red = 41

Most other languages have support for naming enumerations
without explicitly declaring their values (albeit with
specialized syntax that makes it cleaner):

C++: http://en.cppreference.com/w/cpp/language/enum
C#: https://msdn.microsoft.com/en-us/library/sbbt4032.aspx
Java: https://docs.oracle.com/javase/tutorial/java/javaOO/enum.html
Rust: https://doc.rust-lang.org/book/enums.html

Currently, a developer can copy the code from the Python docs into
his or her project, or add a dependency on aenum. I would argue
that it belongs in the standard library.

If there are objections to it being too magical, I would argue
it's already spelled out in the docs, so it's being prescribed
as a solution already. I think it should be very clear when you
derive from "AutoNumberedEnum" what is going on.

The code is very simple, so I would
hope maintenance would not be difficult.

On the other hand, if you use aenum you can do:

class Color(AutoNumber):
    red
    green
    blue

And isn't that better*? ;)

For those in danger of swallowing their tongue over the magic involved: The magic is turned off as soon as any descriptor is defined (property, function, etc.).

Wow Ethan, that's quite interesting. I'm not sure if I like it or not. :)

It's better than having to assign a dummy value to the enum items, and if we did require that, I'd suggest just throwing away the values for autonumbering.

But not having to specify any values at all is probably better, although it does look weird!

/me goes to look at the magic you're using...

It's in _EnumDict.__getitem__; there's some duplication in __setitem__ for supporting Python 2 (although with 2 you have to use either the empty tuple, or some other handy thing that may go in the __doc__ attribute, for example).

@Ethan

@Ethan, I hadn't tried to use the aenum AutoNumberEnum that way, but I agree with Barry that I like it. To me that is the ideal case we should shoot for as I think it's the best* long term and deviate only if practical concerns prevent it.

So I am +1 for empty member assignment and if that is rejected, +1 for = () assignment as at least it is a big step forward. I feel both solutions already have some "magic", so would lean toward the one that leads to the least amount of boilerplate.

As for the empty assignment, I have played around with something similar before and will throw out one con for it: static analyzers get really confused. PyCharm, for example, thinks this is has unresolved references in it:

class Color(AutoNumberEnum):
    red
    green
    blue

But the counter point is that if this is in the stdlib, static analyzer authors are much more likely to add a special case for it than if in a "non-official" third party package (PyCharm example: https://youtrack.jetbrains.com/issue/PY-19150).

Another piece of evidence to support inclusion is that Python already provides specialized Enum subclasses (like IntEnum). I've written a lot of Python code that uses Enums and haven't personally needed IntEnum yet, but would have used an AutoEnum many, many times.

I am assuming here that a true "enum" keyword is out of the question at this point for Python, which would probably be even better.

@Ethan/Barry what needs to be done now to accept or reject this for Python 3.6? Should I propose it onto python-dev? If accepted, would be nice to get it in before alpha 3 (~2 weeks). What's nice about this proposal is Ethan has already written the code, so it's just a matter of getting consensus to add it.

I don't really think much more needs to be done other than agree on this tracker issue that it's something we want, and that Ethan's implementation is good enough. It's been a while since I looked that the code but it seemed good to me before, so if Ethan is happy with it going into the stdlib, so am I. As for the feature, it *is* a little magical, but not so much that it would be confusing or ambiguous, so I'm +1 on it.

I'll defer to Ethan for the final decision, but if we have tests and documentation, I'd be very happy to see this in 3.6.

Is this something we want to get in before the next alpha in two days? Just wanted to bring up the deadline since this may be a feature people want to play around with during the alpha phase.

Ethan, I'm happy to help with documentation or anything else.

There is one wrinkle with auto-numbering. Consider the following code:

class Color(AutoNumberEnum):
  red
  green
  blue
  @property
  def cap_name(self):
    return self.name.upper()

What happens with property?

property is looked up in the class namespace
it's not found, so is given the number 4
it's then called to handle cap_name
an exception is raised

As far as I can tell there is only one "good" way around this:

store any names you don't want auto-numbered into an _ignore_
attribute (this only needs to be global and built-in names that
are used before the first method is defined)

another option is to build a proxy around any found global/built-in objects and decide what to do based on whether those objects are immediately called, but that fails when the object is simply assigned for later use.

So, barring any other ideas to handle this problem, the above example should look like this:

class Color(AutoNumberEnum):
  _ignore_ = 'property'
  red
  green
  blue
  @property
  def cap_name(self):
    return self.name.upper()

Another advantage of using ignore is the ability to have temporary variables automatically discarded:

  class Period(timedelta, Enum):
    '''
    different lengths of time
    '''
    _ignore_ = 'Period i'
    Period = vars()
    for i in range(31):
      Period['day_%d' % i] = i, 'day'
    for i in range(15):
      Period['week_%d' % i] = i*7, 'week'
    for i in range(12):
      Period['month_%d' % i] = i*30, 'month'
    OneDay = day_1
    OneWeek = week_1
    def __new__(self, value, period):
      ...

and the final enumeration does not have the temp variables Period nor i.

Thoughts?

On Jul 09, 2016, at 03:20 PM, Ethan Furman wrote:

As far as I can tell there is only one "good" way around this:

store any names you don't want auto-numbered into an _ignore_
attribute (this only needs to be global and built-in names that
are used before the first method is defined)

Seems reasonable, though why not a dunder, e.g. __ignore__?

_ignore_ = 'Period i'

Why space separated names inside a string instead of a sequence of strings?

A standard feature of Enum is that either space separated strings or a list of strings is accepted any where either is.

_sunder_ names are the ones reserved for Enum use (such as _value_, _name_, and soon _order_ (for py2/py3 compatible code).

@staticmethod

What happens with property?

property is looked up in the class namespace

Perhaps you've already considered this, I'm not intimately familiar with how classes are parsed and constructed but is it possible to determine if the object is a decorator? It already determines to stop auto-numbering when it hits the first method, could it stop when it hits the first decorator or method?

Being able to use temporaries is an interesting side effect, but I feel like that would be used less often than a @staticmethod, @Property, or @classmethod over a method, in which case it becomes a little more complex.

That being said, I think either solution is valid.

The problem with testing the type of object a name refers to outside the class is it then becomes more difficult to make that an Enum member:

class AddressType(Enum):
    pobox
    mailbox  # third-party po box
    property

Having to assign a value to property pretty much negates the value of the magic.

I'll go with _ignore_.

Would it be possible to use a auto or AUTO_ENUM constant from the enum in
the Enum class? (And drop AutoEnum class.)

I must say I never understood how exactly is assigning a constant better in this regard. To me, after

    a = SOMETHING
    b = SOMETHING

a and b should be the same, right? _Especially_ when SOMETHING is None or (), but also when it's some bare identifier like AUTO_ENUM.

Mental models are already broken. We are already "lying" according to semantics of type metaclass. There is no reason then to have to sprinkle some magic dust over our code, just so it looks like an assignment is the most important thing here, when it really isn't.

Either we should embrace Python's power in full and communicate to users that there's something weird going on, or we shouldn't have AutoEnum. To me, green = None is in no way better, in fact it seems more dishonest, than simply green. Again, if we communicate (e.g. via style='declarative') that the semantics is not ordinary. (Alternatively, we might just write green = uuid4() and be done with it.:)

Explicit is better than implicit :)

Absolutely. How the hell is green = None explicit?!

On the other hand, `class Color(Enum, style='declarative'):` is explicit. People must learn something. Why then don't they learn the right thing instead of "hey, assigning None inside enums magically does the right thing" - except when it doesn't.

Just a hint of a nightmare scenario: you write a method decorator, but you forget to return a value from it (happened to me more times than I care to admit). Ooops, your method is now next member of your enum. Good luck debugging that. :-O

In fact, what _is_ explicit, is this:

    class Color(metaclass=Enum):
        green = object()
        yellow = object()

I could live with it. :-)

2016-08-19 14:19 GMT+02:00 Vedran Čačić <report@bugs.python.org>:

In fact, what _is_ explicit, is this:
class Color(metaclass=Enum):
    green = object()
    yellow = object()
I could live with it. :-)

For me, it's legit to use a singleton in an enum. I mean that I expect
that green != yellow. Some remark for an empty tuple.

So I woud really prefer a contant from enum like AUTO_ENUM.

For me, it's legit to use a singleton in an enum.

But you cannot explain _why_, right? Like Raymond says, it's not working like anything else in Python works.

To me, this looks almost the same as the gripes of those lazy people who want "myfile.close" expression to close myfile, or worse, "quit" to exit the Python interpreter. An irrational phobia of parentheses. Why not call your object, if you expect some code to be executed? [At least a dotted name like myfile.close _could_ be made to work using descriptors, but still I think everyone agrees it's not a good idea.]

To me, we either want to stay with default type metaclass (style='imperative':), and write something like

    class Color(Enum):
        green = member()
        yellow = member()

or we acknowledge that style='declarative' has its place under the sun (not only for Enums, of course... database models, namedtuples, even ABCs could profit from it), and embrace its full power, realizing it's not Python we usually see, but it's still Python.

Magic is not something to be afraid of, if you understand it. We are talking Py3.6 here... aren't formatted strings a totally insane magic? Yet they ended up in the language, because they are immensely better than the alternatives. Here the gain is much smaller, but the threshold is much lower too: we don't need new syntax at all.

Vedran, you have some very interesting points. However, if you really want to champion this you'll need to take it to Python Ideas.

Hey, I just realized that you can get pretty darn close with just a little bit
of extra typing, and existing stdlib:

    from enum import Enum
    from itertools import count

    auto = count()

    class Color(Enum):
        red = next(auto)
        green = next(auto)
        blue = next(auto)

Look ma, no magic!

No magic, but a whole heap of extra boiler-plate. :(

There's a difference between boiler-plate and "code".

On Aug 19, 2016, at 10:53 AM, Ethan Furman <report@bugs.python.org> wrote:

Ethan Furman added the comment:

No magic, but a whole heap of extra boiler-plate. :(

----------

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue26988\>

One solution similar to one proposed by Vedran works with the current Enum:

    class Color(Enum):
        red = object()
        green = object()
        blue= object()

I tested this in PyCharm and it is perfectly happy with the autocomplete and everything. The main disadvantage is the boilerplate, of course. And perhaps "object()" does not show the clearest intent, but it depends on your perspective. The repr also looks kind of funny:

    >>>  repr(Color.red)
    <Color.red: <object object at 0x7fb2f353a0d0>>

One possibility would be to add an auto() function to enum as a wrapper around object(), providing a more explicit name and a cleaner repr:

    from enum import Enum, auto

    class Color(Enum):
        red = auto()
        blue = auto()
        green = auto()

    repr(Color.red)
    <Color.red>
    # auto() means it has no (meaningful) value, so show nothing

# Just wait until I put the keys to the time machine in their usual place... :-)

Ok, while we're talking about whether declarative style is a good idea, Python has already got the initial seed of that style. With Guido's blessing! PEP-526 enables us to mix declarative and imperative style in "ordinary" code, so we can write

    @Enum
    class Color:
        green: member
        yellow: member

without any additional syntax. I think it satisfies everyone: there are no parentheses, and there are no assignments. [And there is no misleading analogy with existing syntax, because this is a new syntax.:] There are just declarations, and the decorator instantiates them.

Decorator is needed because formally we need to exclude the type checking semantics, and the only official way currently is through a decorator. But in fact we can use the forward references to _actually_ annotate the members with their real type:

    class Color(Enum):
        green: 'Color'
        yellow: 'Color'

And once the forward references get a nicer syntax, and the unpacking issues are solved, we'll be able to write

    class Color(Enum):
        green, yellow: Color

And I think finally everyone will be happy. :-)

New changeset 2e243f78720e by Ethan Furman in branch 'default':
bpo-26988: remove AutoEnum
https://hg.python.org/cpython/rev/2e243f78720e

Navigation Menu

Repository navigation

Add AutoNumberedEnum to stdlib #71175

39 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Add AutoNumberedEnum to stdlib #71175

Description

Activity

johnthagen commented on May 10, 2016

ethanfurman commented on May 10, 2016

warsaw commented on May 10, 2016

ethanfurman commented on May 10, 2016

johnthagen commented on May 10, 2016

johnthagen commented on Jun 29, 2016

warsaw commented on Jun 29, 2016

johnthagen commented on Jul 9, 2016

ethanfurman commented on Jul 9, 2016

warsaw commented on Jul 9, 2016

ethanfurman commented on Jul 9, 2016

johnthagen commented on Jul 9, 2016

ethanfurman commented on Jul 9, 2016

39 remaining items

vstinner commented on Aug 19, 2016

vedgar commented on Aug 19, 2016

kennethreitz commented on Aug 19, 2016

vedgar commented on Aug 19, 2016

vstinner commented on Aug 19, 2016

vedgar commented on Aug 19, 2016

ethanfurman commented on Aug 19, 2016

warsaw commented on Aug 19, 2016

ethanfurman commented on Aug 19, 2016

kennethreitz commented on Aug 19, 2016

DavidHagen commented on Aug 19, 2016

vedgar commented on Aug 19, 2016

python-dev commented on Aug 20, 2016

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions