Skip to content

enum.IntFlag regression: missing values cause TypeError #88408

Closed
Closed
enum.IntFlag regression: missing values cause TypeError#88408
Assignees
ethanfurman
Labels
3.10 (EOL)end of life3.11only security fixesstdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

Description

@hroncok
hroncok
mannequin
opened on May 26, 2021
BPO 44242
Nosy @belm0, @ethanfurman, @vedgar, @tacaswell, @hroncok, @pablogsal, @miss-islington, @Zheaoli, @belm0, @hauntsaninja, @jacobtylerwalls
PRs
  • bpo-44242: [Enum] remove missing bits test from Flag creation #26586
  • [3.10] bpo-44242: [Enum] remove missing bits test from Flag creation (GH-26586) #26635
  • bpo-44242: [Enum] improve error messages #26669
  • [3.10] bpo-44242: [Enum] improve error messages (GH-26669) #26671
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/ethanfurman'
closed_at = <Date 2021-06-14.19:05:28.255>
created_at = <Date 2021-05-26.19:31:43.243>
labels = ['type-bug', 'library', '3.10', '3.11']
title = 'enum.IntFlag regression: missing values cause TypeError'
updated_at = <Date 2021-06-14.19:05:28.255>
user = 'https://github.com/hroncok'

    bugs.python.org fields:

    activity = <Date 2021-06-14.19:05:28.255>
actor = 'ethan.furman'
assignee = 'ethan.furman'
closed = True
closed_date = <Date 2021-06-14.19:05:28.255>
closer = 'ethan.furman'
components = ['Library (Lib)']
creation = <Date 2021-05-26.19:31:43.243>
creator = 'hroncok'
dependencies = []
files = []
hgrepos = []
issue_num = 44242
keywords = ['patch']
message_count = 27.0
messages = ['394457', '394461', '394464', '394466', '394467', '394468', '394469', '394470', '394471', '394472', '394473', '394475', '394479', '394486', '394487', '394500', '394593', '395134', '395136', '395431', '395539', '395572', '395616', '395628', '395629', '395810', '395838']
nosy_count = 11.0
nosy_names = ['jbelmonte', 'ethan.furman', 'veky', 'tcaswell', 'hroncok', 'pablogsal', 'miss-islington', 'Manjusaka', 'John Belmonte', 'hauntsaninja', 'jacobtylerwalls']
pr_nums = ['26586', '26635', '26669', '26671']
priority = None
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue44242'
versions = ['Python 3.10', 'Python 3.11']

    Activity

    1. hroncok commented on May 26, 2021

      @hroncok
      hroncokmannequin
      on May 26, 2021
      MannequinAuthor

      With the change introduced in https://bugs.python.org/issue38250 7aaeb2a I observe a regression in behavior of enum.IntFlag with missing values.

      Consider this code (from pyproj):

          from enum import IntFlag

    class GeodIntermediateFlag(IntFlag):
        DEFAULT = 0x0

        NPTS_MASK = 0xF
        NPTS_ROUND = 0x0
        NPTS_CEIL = 0x1
        NPTS_TRUNC = 0x2
         
        DEL_S_MASK = 0xF0
        DEL_S_RECALC = 0x00
        DEL_S_NO_RECALC = 0x10
      
        AZIS_MASK = 0xF00
        AZIS_DISCARD = 0x000
        AZIS_KEEP = 0x100

      This is a valid code in Python 3.9, however produces a TypeError in Python 3.10.0b1:

          Traceback (most recent call last):
      File "intflag.py", line 3, in <module>
        class GeodIntermediateFlag(IntFlag):
      File "/usr/lib64/python3.10/enum.py", line 544, in __new__
        raise TypeError(
    TypeError: invalid Flag 'GeodIntermediateFlag' -- missing values: 4, 8, 32, 64, 128, 512, 1024, 2048

      Since I don't see this behavior mentioned in https://docs.python.org/3.10/library/enum.html or https://docs.python.org/3.10/whatsnew/3.10.html or https://docs.python.org/3.10/whatsnew/changelog.html I believe this is a regression.

    2. hroncok
      added
      3.10 (EOL)end of life
      stdlibStandard Library Python modules in the Lib/ directory
      type-bugAn unexpected behavior, bug, or error
      on May 26, 2021

    3. pablogsal commented on May 26, 2021

      @pablogsal
      pablogsal
      on May 26, 2021
      Member

      We are already blocked in Python 3.10 beta 2, Ethan, could you give a look at this so we can introduce the fix when we release the next beta?

    4. ethanfurman commented on May 26, 2021

      @ethanfurman
      ethanfurman
      on May 26, 2021
      Member

      That is an intentional change. The cause is that the masks include bits that are not named in the Flag.

      The user-side fix is to add a boundary=KEEP option to the flag:

          class GeodIntermediateFlag(IntFlag, boundary=KEEP)

      The enum library fix could be one of two things:

      • automatically use the KEEP boundary when these conditions arise, and issue a DeprecationWarning; or

      • lose that particular check.

      I'm inclined to go with option 2, since boundary is designed to answer the question of what to do when Flag.A | Flag.B does not exist in Flag.

    5. belm0 commented on May 26, 2021

      @belm0
      belm0mannequin
      on May 26, 2021
      Mannequin

      To clarify, it's caused by these mask entries in the enum:

      NPTS_MASK = 0xF
DEL_S_MASK = 0xF0
AZIS_MASK = 0xF00

      Since the masks are not an aggregation of individual bits defined in the enum, it's an error.

      I'm inclined to go with [removing the check], since boundary is designed to answer the question of what to do when Flag.A | Flag.B does not exist in Flag.

      I think that would cause various problems in the API and implementation. For example, without underlying individual bits, repr() may be nonsensical.

    6. belm0 commented on May 26, 2021

      @belm0
      belm0mannequin
      on May 26, 2021
      Mannequin

      I wonder if CONFORM could tolerate these, since it's ultimately going to discard invalid bits. And then perhaps CONFORM is the default.

    7. ethanfurman commented on May 26, 2021

      @ethanfurman
      ethanfurman
      on May 26, 2021
      Member

      Actually, thinking about that a little bit more, KEEP was added for exactly this situation, as some stdlib flags exhibit the same behavior.

      So the real question is what should happen with, for example,

        GeodIntermediateFlag(0x80)

      ?

      The idea behind boundary is what should happen when values are created that don't have names in the Enum/Flag? The options for boundary are:

      STRICT -> an error is raised (default for Enum)
      EJECT -> the integer 0x80 is returned (not a flag)
      CONFORM -> unnamed bits are discarded (so the DEFAULT flag would be returned)
      KEEP -> an unnamed flag with value 0x80 is returned

      So KEEP is currently doing double-duty -- this reinforces my desire to go with option 2 and return KEEP to single-duty status.

    8. ethanfurman commented on May 26, 2021

      @ethanfurman
      ethanfurman
      on May 26, 2021
      Member

      Those are good points -- the difficulty is knowing which behavior the user wants. And if the desired run-time behavior doesn't match the boundary flag the user is stuck.

    9. ethanfurman commented on May 26, 2021

      @ethanfurman
      ethanfurman
      on May 26, 2021
      Member

      For example, if the default is CONFORM or KEEP, but the user wants an error if 0x80 comes up, they would have to explicitly check for that value since the Flag would happily return it instead of raising.

    10. belm0 commented on May 26, 2021

      @belm0
      belm0mannequin
      on May 26, 2021
      Mannequin

      Rather than make such masks containing unknown bits, this would be best practice if you want to use STRICT, correct?

      NPTS_ROUND = 0x0
NPTS_CEIL = 0x1
NPTS_TRUNC = 0x2
NPTS_MASK = NPTS_ROUND | NPTS_CEIL | NPTS_TRUNC

      Otherwise, if your input may have unknown bits, use CONFORM.

    11. 18 remaining items

    12. ethanfurman
      closed this as completedon Jun 10, 2021
    13. ethanfurman
      added
      3.11only security fixes
      and removed
      release-blocker
      on Jun 10, 2021
    14. ethanfurman
      closed this as completedon Jun 10, 2021
    15. ethanfurman
      removed
      release-blocker
      on Jun 10, 2021

    16. ethanfurman commented on Jun 11, 2021

      @ethanfurman
      ethanfurman
      on Jun 11, 2021
      Member

      Also changing error reporting to be less susceptible to DOS attacks.

    17. ethanfurman
      reopened this on Jun 11, 2021
    18. ethanfurman
      reopened this on Jun 11, 2021

    19. ethanfurman commented on Jun 11, 2021

      @ethanfurman
      ethanfurman
      on Jun 11, 2021
      Member

      New changeset c956734 by Ethan Furman in branch 'main':
      bpo-44242: [Enum] improve error messages (GH-26669)
      c956734

    20. ethanfurman commented on Jun 11, 2021

      @ethanfurman
      ethanfurman
      on Jun 11, 2021
      Member

      New changeset 0a186b1 by Miss Islington (bot) in branch '3.10':
      bpo-44242: [Enum] improve error messages (GH-26669)
      0a186b1

    21. jacobtylerwalls commented on Jun 14, 2021

      @jacobtylerwalls
      jacobtylerwallsmannequin
      on Jun 14, 2021
      Mannequin

      With the followup patch merged, can this be closed now?

    22. ethanfurman commented on Jun 14, 2021

      @ethanfurman
      ethanfurman
      on Jun 14, 2021
      Member

      Yup, just had to get back from the weekend. :-)

    23. ethanfurman
      closed this as completedon Jun 14, 2021
    24. ethanfurman
      closed this as completedon Jun 14, 2021
    25. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    3.10 (EOL)end of life3.11only security fixesstdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions