enum.Flag ~ bitwise negation is very slow and can't be defined as a Flag value

@ethanfurman

BPO	42915
Nosy	@ethanfurman, @aspin
Superseder	bpo-38250: enum.Flag should be more set-like

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ethanfurman'
closed_at = <Date 2021-01-25.23:11:52.419>
created_at = <Date 2021-01-12.23:31:13.479>
labels = ['type-bug', '3.8']
title = "enum.Flag ~ bitwise negation is very slow and can't be defined as a Flag value"
updated_at = <Date 2021-01-25.23:11:52.418>
user = 'https://github.com/aspin'

bugs.python.org fields:

activity = <Date 2021-01-25.23:11:52.418>
actor = 'ethan.furman'
assignee = 'ethan.furman'
closed = True
closed_date = <Date 2021-01-25.23:11:52.419>
closer = 'ethan.furman'
components = []
creation = <Date 2021-01-12.23:31:13.479>
creator = 'aspin2'
dependencies = []
files = []
hgrepos = []
issue_num = 42915
keywords = []
message_count = 5.0
messages = ['384983', '385004', '385054', '385057', '385678']
nosy_count = 2.0
nosy_names = ['ethan.furman', 'aspin2']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = '38250'
type = 'behavior'
url = 'https://bugs.python.org/issue42915'
versions = ['Python 3.8']

Here's a code sample:

import time
from enum import Flag, auto


class MyFlag(Flag):
    NONE = 0
    FLAG_1 = auto()
    FLAG_2 = auto()
    FLAG_3 = auto()
    FLAG_4 = auto()
    FLAG_5 = auto()
    FLAG_6 = auto()
    #
    # NOT_FLAG_1_OR_2 = ~FLAG_1 & ~FLAG_2


def test_flag():
    f = MyFlag.NONE
    inverted = (
        ~MyFlag.FLAG_1
        & ~MyFlag.FLAG_2
        & ~MyFlag.FLAG_3
        & ~MyFlag.FLAG_4
        & ~MyFlag.FLAG_5
        & ~MyFlag.FLAG_6
    )
    return f & inverted


INVERTED = (
    ~MyFlag.FLAG_1
    & ~MyFlag.FLAG_2
    & ~MyFlag.FLAG_3
    & ~MyFlag.FLAG_4
    & ~MyFlag.FLAG_5
    & ~MyFlag.FLAG_6
)


def test_flag_cached():
    f = MyFlag.NONE
    return f & INVERTED


if __name__ == "__main__":
    start_time = time.time()
    for _ in range(10_000):
        test_flag()

    elapsed = time.time() - start_time
    print(f"Took normal {elapsed:2f} seconds.")

    start_time = time.time()
    for _ in range(10_000):
        test_flag_cached()

    elapsed = time.time() - start_time
    print(f"Took cached {elapsed:2f} seconds.")

And its outputs:

Took normal 1.799731 seconds.
Took cached 0.009488 seconds.

Basically, bitwise negation is very very slow. From what I can tell, it seems that a lot of time is spent here computing powers of two. I've read elsewhere that flag values are cached, and it looks like negated Flag values can't be cached? This seems related to the second issue, which is that any negated Flag value being defined results in RecursionError: maximum recursion depth exceeded as it searches for a good name for Flag.

Obviously, the simple workaround is just to define a constant variable elsewhere with the negated value, but it isn't very obvious anywhere that this is necessary, and I wanted to raise this to see if anyone has knowledge of the implementation details of Flag for possibly resolving this in the class itself.

I just finished a rewrite of Flag for 3.10. Using your test below I was able to tweak the rewrite so the final numbers are:

Took normal 0.148092 seconds.
Took cached 0.017438 seconds.

Your original post had a ratio of nearly 200 -- it is now 8.7ish.

Awesome thanks! Does the rewrite fix the issue with creating negated flags as well?

Yes.

Fixed in 3.10 in bpo-38250.

Also fixed in my 3rd-party library, aenum 3.0:

(https://pypi.org/project/aenum/)

Navigation Menu

Repository navigation

enum.Flag ~ bitwise negation is very slow and can't be defined as a Flag value #87081

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

enum.Flag ~ bitwise negation is very slow and can't be defined as a Flag value #87081

Description

Activity

aspin commented on Jan 12, 2021

ethanfurman commented on Jan 13, 2021

aspin commented on Jan 13, 2021

ethanfurman commented on Jan 13, 2021

ethanfurman commented on Jan 25, 2021

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions