enum: Add Flags and IntFlags

@warsaw

BPO	23591
Nosy	@warsaw, @rhettinger, @ezio-melotti, @bitdancer, @ethanfurman, @serhiy-storchaka, @MojoVampire, @vedgar
Files	intflags.patch intflags_2.patch intflags_3.patch issue23591.stoneleaf.02.patch bit_length.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ethanfurman'
closed_at = <Date 2016-11-21.16:31:14.558>
created_at = <Date 2015-03-05.15:31:23.274>
labels = ['type-feature', 'library']
title = 'enum: Add Flags and IntFlags'
updated_at = <Date 2016-11-21.16:31:14.556>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2016-11-21.16:31:14.556>
actor = 'python-dev'
assignee = 'ethan.furman'
closed = True
closed_date = <Date 2016-11-21.16:31:14.558>
closer = 'python-dev'
components = ['Library (Lib)']
creation = <Date 2015-03-05.15:31:23.274>
creator = 'serhiy.storchaka'
dependencies = []
files = ['38344', '38387', '44114', '44278', '44319']
hgrepos = []
issue_num = 23591
keywords = ['patch']
message_count = 80.0
messages = ['237269', '237491', '237494', '237532', '247018', '247026', '247033', '247040', '269018', '269023', '269792', '269919', '270110', '272202', '272230', '272303', '272651', '272674', '272693', '272701', '272719', '272732', '272737', '272741', '272788', '272793', '272795', '272796', '272797', '272798', '272799', '272926', '272927', '272956', '272958', '273934', '273939', '273948', '273954', '273958', '274003', '274048', '274105', '274107', '274206', '274230', '274241', '274279', '274280', '274311', '274317', '274343', '274372', '274373', '274377', '274379', '274403', '274822', '274826', '274828', '274868', '274869', '274870', '274875', '274876', '274975', '274976', '274979', '275093', '275736', '275743', '275801', '275802', '276335', '276348', '276354', '276358', '276907', '277432', '281376']
nosy_count = 10.0
nosy_names = ['barry', 'rhettinger', 'ezio.melotti', 'r.david.murray', 'eli.bendersky', 'ethan.furman', 'python-dev', 'serhiy.storchaka', 'josh.r', 'veky']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue23591'
versions = ['Python 3.6']

Here is preliminary implementation of IntFlags (no docs). This is an int subclass purposed to represent a set of integer flags. It supports named constants (as IntEnum), supports bitwise operations (&, |, ~) and has funny str and repr. See discussion on Python-Ideas [1].

The patch includes tests and few examples of using IntFlags in the stdlib.

[1] http://comments.gmane.org/gmane.comp.python.ideas/32267

It would make more sense and be more consistent if the str() and repr() used one’s complement in all cases, i.e.:

self.assertEqual(str(Perm(~0)), "~0")

Also, the repr() seems to be doing a bad attempt at Python pseudo code. Instead of

<Perm.R|W: 3>

maybe it could be something like these:

<Perm.R|Perm.W: 3> # Mirroring str() -> "Perm.R|Perm.W"
<Perm R|W: 3>
<Perm: R|W = 3>

I wonder if the addition (+) operator should also be overridden; I haven’t looked, but I suspect some people may do FLAG1 + FLAG2 instead of FLAG1 | FLAG2.

The current patch is more along the lines of a proof-of-concept.

The final IntFlag type (if there is one) would be quite a bit more extensive since part of the reason for its existence is to not lose type -- so pretty much every __op__ would have to be overridden.

Agreed about the repr() and str(), though.

I choose repr() so that for single flag it looks as for IntEnum instance. Other variants except <Perm.R|Perm.W: 3> have different format. <Perm.R|Perm.W: 3> is too verbose, it repeats the same class name multiple times.

But I don't like current repr.

Other operators are not overridden for purpose. IntFlags is int and can be used in all cases when int is used, but not in all cases the result should preserve the type. FLAG1 + FLAG2 works as if flags would be ints, but it should be rewritten to FLAG1 | FLAG2 to got the benefit from using IntFlags.

However I forgot to override the exclusive-or operator. It should be overridden.

Updated patch changes str and repr, override the exclusive-or operator and adds some documentation.

Why have you rejected this issue Ethan? I think this is useful feature and must be in Python, and other core developers agreed with this. Only minor implementation details are discussable.

I agree. Closing this does not follow our normal development workflow, since there has been a consensus in favor and none for rejection. If you think the API needs wider discussion a new thread can be started on python-ideas.

Ethan: to clarify...based on my years of watching this tracker, the way we normally handle an issue like this is to leave the issue open until it is resolved, sometimes with people proposing competing patches. (There might be reasons to change that tradition, but if so the forum for that discussion would be python-workflow.)

My experience is that a module maintainer, or somebody claiming to speak for the module maintainer, can close any issue in their area at any time regardless of the number of core devs in favor of a change.

Whatever. I'll leave this open and write up a spec of what IntFlags should look like in case somebody wants to write the patch for it.

What should be done for landing this in 3.6?

Just look briefly through your patches, and they look pretty good. I'll take a more in-depth look in the next couple weeks. (Feel free to ping again if you don't see any activity from me, and thanks for your patience.)

New changeset aad7443e62be by Ethan Furman in branch 'default':
bpo-23591: add auto() for auto-generating Enum member values
https://hg.python.org/cpython/rev/aad7443e62be

Which means it has methods such as __getitem__, __setitem__, etc., which means those methods can implement whatever is needed to give the namespace the desired semantics (within Python syntax).

Ah, _that_'s what you had in mind. (All this time, I thought _auto_ was just a shorter name for _generate_next_value_.:) I'm ok with that. But aren't we then back to square one? (Using magic of the same kind as the "declarative style" that Raymond pronounced not Pythonic enough.)

Even Raymond says

As long as _auto_ has been defined somewhere (i.e. from enum import _auto_), it is normal Python

I'm sure he didn't think of the loophole you're currently exploiting. :-D

But anyway, I'm happy. Parentheses are there, so the intuition of calling a function to execute code is respected, and if you get out free with this hack, it opens a door a bit wider for complete declarative solution in the future.

> Which means it has methods such as __getitem__, __setitem__, etc.,
> which means those methods can implement whatever is needed to give
> the namespace the desired semantics (within Python syntax).

Ah, _that_'s what you had in mind. (All this time, I thought _auto_
was just a shorter name for _generate_next_value_.:) I'm ok with that.
But aren't we then back to square one? (Using magic of the same kind
as the "declarative style" that Raymond pronounced not Pythonic enough.)

Not at all. The concern with that proposal (bpo-26988: AutoEnum via completely omitting values of any kind) was the transformation of a NameError into a value. The current method is taking a unique object and transforming that into a value, which is entirely analagous to what Enum already does.

Besides being convenient, it's also necessary to present a cohesive, non-leaky abstraction to the user where Flag is concerned: using ints as the flag values is entirely an implementation detail, but without auto the user would be forced to manage that implementation detail when creating the Flag, which negates much of Flag's value.

Many thanks to everyone for their input and help.

Now that I actually had the chance to play with the implementation, I see most of my worst fears were justified. :-( Look:

    >>> import enum
    >>> class B(enum.Flag):
            b = 3
            c = 4
            d = 6
    >>> B.b | B.c
    Traceback (most recent call last): ...
    ValueError: 7 is not a valid B
    >>> t = B.b | B.c
    >>> t
    <B.d|1: 7>
    >>> B.b | B.c
    >>> B.b | B.c
    <B.d|1: 7>
    >>> ~B.d
    <B.0: 0>

Do you really find this behavior acceptable?? I see at least three bugs here.

At least you did say in the documentation

Individual flags should have values that are powers of two (1, 2, 4, 8, ...)

but it seems to me it's not prominent enough.

---

Also, auto, despite parentheses, works exactly as it shouldn't.

    >>> class C(enum.Enum):
	a = b = enum.auto()	
    >>> C.a
    <C.a: 1>
    >>> C.b
    <C.b: 2>

    >>> def f():
            return enum.auto()
    >>> class E(enum.Enum):
            a = b = f()
            c = d = 2
    >>> E.a is E.b
    False
    >>> E.c is E.d
    True
    >>> E.b is E.c
    True

In my opinion, this is simply horrible. Even _you_ said (http://bugs.python.org/issue23591#msg275093) this would be unacceptable. I'm really, really sad.

Vedran:

Thank you for the bug report, it's appreciated.
I would appreciate it even more if you just kept it to a bug report
without the, for lack of a better word, drama.

-----------------------------------------------

>    >>> import enum
>    >>> class B(enum.Flag):
>            b = 3
>            c = 4
>            d = 6
>    >>> B.b | B.c
>    Traceback (most recent call last): ...
>    ValueError: 7 is not a valid B

This part is correct. If the programmer wants to use actual values instead of auto() then it is on them to do it correctly. I'd be happy to add a decorator to help ensure all bits have names, but I'm not sure what to call it.

-----------------------------------------------

>>> t = B.b | B.c
>>> t
<B.d|1: 7>

>>> ~B.d
<B.0: 0>

These parts are not, and will be fixed.

-----------------------------------------------

>>> class C(enum.Enum):
a = b = enum.auto()
>>> C.a
<C.a: 1>
>>> C.b
<C.b: 2>

Bug, will be fixed.

>>> def f():
return enum.auto()
>>> class E(enum.Enum):
a = b = f()
c = d = 2
>>> E.a is E.b
False

Same as above bug, will be fixed

>>> E.c is E.d
True
>>> E.b is E.c
True

Same as above bug, will be fixed. (Had "a" and "b" been given different "auto()"s that last True would be correct as "b" would then have had a value of 2 from auto(), and "c" and "d" also were given values of 2, so all three would have been the same.)

About drama: I don't know how to approach the problem. I tell you about various concerns of mine, you seem to agree, and still you do exactly what I was warning you about. It could be that you just consider me a boring nag, but then saying so would be more useful than this situation.

---

Yes, you solved some problems I did have, and it's great, but still many of them remain. This thing is inherently complicated. You are trying to construct a discrete Boolean algebra, but without any guarantee of atoms. Imagine if the language was different:

    class MySets(DiscreteSets):
        a = {0, 1}
        b = {2}
        c = {1, 2}

You'd naturally view {0}, {1} and {2} as atoms, of different "sort" than {0,1} and such. That's what's missing here: a clear distinction between atoms and "molecules". It could be that the intended usage is for atoms to _always_ be constructed with auto(), and for molecules to be constructed from previous atoms, but then the documentation ought to say so.

So far, the only thing you say is "then it is on them to do it correctly", but you never really tell what "correctly" means. I understand your desire not to hard forbid anything, but the docs should tell how your class is _intended_ to be used, _AND_ the class should make every reasonable effort to preserve what it can of its behavior even in the face of not-quite-intended usage. Look at Raymond's collections.Counter's treatment of non-integer values for example.

BTW I'm not aware of any stdlib object whose repr raises ValueError. It surely makes debugging a lot harder.

I'll respond to the bulk of your comment later. For now let me assure you your feedback has been invaluable.

New changeset b56290a80ff7 by Ethan Furman in branch '3.6':
bpo-23591: fix flag decomposition and repr
https://hg.python.org/cpython/rev/b56290a80ff7

New changeset 7372c042e9a1 by Ethan Furman in branch 'default':
bpo-23591: fix flag decomposition and repr
https://hg.python.org/cpython/rev/7372c042e9a1

The "What’s New In Python 3.6" page should really get an update to mention Flag, IntFlag and auto as enhancements to the enum module. Right now, the ssl module update docs mentions (but does not properly link for some reason) IntFlag, but otherwise, there is no mention of the new feature in either the "What's New" docs or even in the theoretically comprehensive changelog; if you don't think to look at the enum module itself, you'd never know there were new features to use.

New changeset f404a59d834e by Ethan Furman in branch '3.6':
closes bpo-23591: add NEWS entry
https://hg.python.org/cpython/rev/f404a59d834e

Navigation Menu

Repository navigation

enum: Add Flags and IntFlags #67779

61 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

enum: Add Flags and IntFlags #67779

Description

Activity

serhiy-storchaka commented on Mar 5, 2015

vadmium commented on Mar 8, 2015

ethanfurman commented on Mar 8, 2015

serhiy-storchaka commented on Mar 8, 2015

serhiy-storchaka commented on Jul 21, 2015

bitdancer commented on Jul 21, 2015

bitdancer commented on Jul 21, 2015

ethanfurman commented on Jul 21, 2015

serhiy-storchaka commented on Jun 21, 2016

ethanfurman commented on Jun 21, 2016

61 remaining items

python-dev commented on Sep 11, 2016

vedgar commented on Sep 11, 2016

ethanfurman commented on Sep 11, 2016

ethanfurman commented on Sep 11, 2016

vedgar commented on Sep 13, 2016

ethanfurman commented on Sep 13, 2016

vedgar commented on Sep 13, 2016

ethanfurman commented on Sep 13, 2016

python-dev commented on Sep 18, 2016

MojoVampire commented on Sep 26, 2016

python-dev commented on Nov 21, 2016

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions