ForwardRef.__eq__ does not respect module parameter #90491

Activity

1. aha79 commented on Jan 10, 2022

   

   aha79mannequin

   on Jan 10, 2022

   MannequinAuthor

   More actions

   The __eq__ method of ForwardRef does not take into account the module parameter.

   However, ForwardRefs with dissimilar module parameters are referring to different types even if they have different name. Thus also the ForwardRef's with same name but different module, should not be considered equal.

   Consider the following code

   from typing import *
   
   ZZ = Optional['YY'] 
   YY = int
   
   YY = Tuple[Optional[ForwardRef("YY", module=__name__)], int]
   print( YY.__args__[0].__args__[0].__forward_module__ )
   # this prints None, but should print __main__ (or whatever __name__ contains)

   When the first ForwardRef is not created, the program behaves correctly

   #ZZ = Optional['YY'] 
   YY = int
   
   YY = Tuple[Optional[ForwardRef("YY", module=__name__)], int]
   print( YY.__args__[0].__args__[0].__forward_module__ )
   # this prints __main__ (or whatever __name__ contains)

   The issue is that the line ZZ = Optional['YY'] creates a cache entry, which is re-used instead of the second definition Optional[ForwardRef("YY", module=__name__)] and thus shadows the different argument of ForwardRef.

   This problem could be fixed if the __eq__ method of FowardRef also checks for module equality.

   i.e. in ForwardRef.__eq__ in typing.py replace

   return self.__forward_arg__ == other.__forward_arg__

   with

   return self.__forward_arg__ == other.__forward_arg__ and self.__forward__module__ == other.__forward__module__

   Ideally, (and for consistency reasons) the __repr__ method of ForwardRef would also include the module arg if it present:

   Change:

       def __repr__(self):
           return f'ForwardRef({self.__forward_arg__!r})'

   to

       def __repr__(self):
           if self.__forward_module__ is None:
               return f'ForwardRef({self.__forward_arg__!r})'
           else:
               return f'ForwardRef({self.__forward_arg__!r}, module={self.__forward__module!r})'
2. aha79

   added

   3.10 (EOL)end of lifeend of life

   3.9 (EOL)end of lifeend of life

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Jan 10, 2022
3. gvanrossum commented on Jan 10, 2022

   

   gvanrossum

   on Jan 10, 2022

   Member

   More actions

   Good catch. Do you want to submit a PR? I agree that the repr() could also be better (but please only show other values if they differ from the default).
4. gvanrossum

   added

   3.11only security fixesonly security fixes

   on Jan 10, 2022
5. aha79 commented on Jan 10, 2022

   

   aha79mannequin

   on Jan 10, 2022

   MannequinAuthor

   More actions

   I will give it a try.
6. Fidget-Spinner commented on Jan 12, 2022

   

   Fidget-Spinner

   on Jan 12, 2022

   Member

   More actions

   @andreas
   First of all, thanks for addressing my questions on the PR. I have one more question which I'll post here since it's not entirely related to your PR.

   ForwardRef isn't meant to be explicitly instantiated by a user 1 (it's a typing internal class), so do you mind sharing what your current use case is please? My concern here is that exposing things in repr would force us to keep module forever (I'm 100% fine with eq and hash). After digging up the commit history, I now recall that the module parameter was used solely as a fix for ForwardRefs in different-module TypedDict (see #71204 or bpo-41249). It's not used anywhere else, so I don't know if your current use case was ever intended when the module parameter was added.
7. aha79 commented on Jan 12, 2022

   

   aha79mannequin

   on Jan 12, 2022

   MannequinAuthor

   More actions

   Yeah, sure. The use-case is (de)serialization. Right now I use the library cattr, but there are many others.

   If you are interested there is related discussion in the cattr board 1.

   The original problem is how to define the types for serialization.

   1. If everything is a class, things work well, but not if type aliases are used:

   2. Type aliases sometimes have to be used - they cannot be avoided in all cases,
      especially with recursive types. The famous example is

         Json = Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ]

   (Note: even though mypy does not support this construct, pylance meanwhile does 2)

   3. typing.Annotated seems to be made for specifying additional information such as value ranges, right to be used
      in (de)serialization + validation contexts. Often these will just be type aliases (not used as class members).
      Combination is possible with typing.NewType.

   The problem is that the implicit ForwardRef('Json') cannot be automatically resolved (as it only a name with no context).
   There is really no way this could be handle inside a library such as cattr.

   When one wants to separate interface from implementation this issue is even more complicated. The module where the serialization function is called is typically different from the module with the type definition (This is probably more the norm than the exception)

   The option I expored is to explicitly create the ForwardRef and specify the module parameter (even though I have to admit that I also did read that the ForwardRef is only for internal use).

   Json = Union[ List[ForwardRef(Json',module=__name__)], Dict[str, ForwardRef(Json',module=__name__)], int, float, bool, None ]
   

   Ugly, but this is better than nothing.

   A (worse) alternative is to do

       Json = Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ]
       typing._eval_type(Json, globals(), locals())

   That works since it puts the ForwardRefs into "evaluated" state (self.__forward_value__ is then set)

   A third, but future, alternative could be to automatically decompose the argument of ForwardRef into module and type. Then one could write

      Json = Union[ List[__name__+'.Json'], Dict[str, __name__+'.Json'], int, float, bool, None ]

   and all above problems would be solved.

   Then ForwardRef could stay internal and this is more readable. Even though, the static type checkers would need to understand __name__+'TYPE' constructs to be useful.

   Anyhow, it would be nice to have a solution here.
8. aha79 commented on Jan 12, 2022

   

   aha79mannequin

   on Jan 12, 2022

   MannequinAuthor

   More actions

   Ah, let me add one point: PEP-563 (-> from __future__ import annotations) is also not helping.

   Even with PEP-563 enabled, the JSON example

      Json = Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ]

   needs to be written in exact the same way as without PEP-563. In other words there are cases where ForwardRef cannot be avoided. And unforntunately these are the cases where we have the ForwardRef missing context issue.
9. gvanrossum commented on Feb 12, 2022

   

   gvanrossum

   on Feb 12, 2022

   Member

   More actions

   New changeset b70690b by aha79 in branch 'main':
   bpo-46333: include module in ForwardRef.__repr__ (bpo-31283)
   b70690b
10. JelleZijlstra commented on Feb 17, 2022

    

    JelleZijlstra

    on Feb 17, 2022

    Member

    More actions

    New changeset 6e7b813 by aha79 in branch 'main':
    bpo-46333: Honor module parameter in ForwardRef (GH-30536)
    6e7b813
11. miss-islington commented on Feb 17, 2022

    

    miss-islington

    on Feb 17, 2022

    Contributor

    More actions

    New changeset a17d59a by Miss Islington (bot) in branch '3.10':
    [3.10] bpo-46333: Honor module parameter in ForwardRef (GH-30536) (GH-31379)
    a17d59a
12. miss-islington commented on Feb 17, 2022

    

    miss-islington

    on Feb 17, 2022

    Contributor

    More actions

    New changeset 29ae7d3 by Miss Islington (bot) in branch '3.9':
    bpo-46333: Honor module parameter in ForwardRef (GH-30536)
    29ae7d3
13. JelleZijlstra

    closed this as completedon Feb 17, 2022
14. JelleZijlstra

    closed this as completedon Feb 17, 2022
15. ezio-melotti

    transferred this issue fromon Apr 10, 2022