types.UnionType is not subscriptable #89581

Activity

1. wyfo commented on Oct 9, 2021

   

   wyfomannequin

   on Oct 9, 2021

   MannequinAuthor

   More actions

   types.UnionType is not subscriptable, and this is an issue when type manipulations are done.

   A common maniputation I've to do is to substitute all the TypeVar of a potential generic type by their specialization in the given context.
   For example, given a class:

   @dataclass
   class Foo(Generic[T]):
       bar: list[T]
       baz: T | None

   in the case of Foo[int], I want to compute the effective type of the fields, which will be list[int] and int | None.
   It could be done pretty easily by a recursive function:

   def substitute(tp, type_vars: dict):
       origin, args = get_origin(tp), get_args(tp)
       if isinstance(tp, TypeVar):
           return type_vars.get(tp, tp)
       elif origin is Annotated:
           return Annotated[(substitute(args[0], type_vars), *args[1:])]
       else:
           return origin[tuple(substitute(arg) for arg in args)]  # this line fails for types.UnionType

   And this is not the only manipulation I've to do on generic types. In fact, all my library (apischema) is broken in Python 3.10 because of types.UnionType.
   I've to substitute types.UnionType by typing.Union everywhere to make things work; types.UnionType is just not usable for dynamic manipulations.

   I've read PEP-604 and it doesn't mention if types.UnionType should be subscriptable or not. Is there a reason for not making it subscriptable?
2. wyfo

   added

   3.10 (EOL)end of lifeend of life

   on Oct 9, 2021
3. Fidget-Spinner commented on Oct 10, 2021

   

   Fidget-Spinner

   on Oct 10, 2021

   Member

   More actions

   I don't understand your example, T | None doesn't return a types.Union object, it returns typing.Union/typing.Optional. (I'm assuming this T is the TypeVar in typing). Which *is* subscriptable.

   >>> (T | None)[int].__origin__
   typing.Union

   If you meant to say: why is typing.Union[] allowed, but not types.UnionType[]? That is intentional. types.UnionType is only meant for builtin types. Once you union with *any* type from typing, it will convert to a typing.Union.

   >>> type(int | str)
   <class 'types.UnionType'>
   
   >>> int | str | T
   typing.Union[int, str, ~T]

   If you intend to reconstruct a types.Union from another types.Union, you can do:

   args = get_args(int | str)
   import operator, functools
   functools.reduce(operator.or_, args)

   And guard this code with an isinstance(tp, types.UnionType) check.
4. wyfo commented on Oct 10, 2021

   

   wyfomannequin

   on Oct 10, 2021

   MannequinAuthor

   More actions

   Indeed, sorry, my example was bad. My library was raising at several place, and I've extrapolated about generic substitution.

   I've indeed other substitutions (without TypeVar), and because they were failing, I've assumed that all of my substitutions were failing; I was wrong about generic one.

   For example, if I want to substitute int | Collection[int] to int | list[int], I will have to replace types.UnionType by typing.Union or use reduce, while it was not necessary in 3.9 where I could just write get_origin(tp)[new_args].

   So I'll have to add some if in my code.
5. wyfo

   closed this as completedon Oct 10, 2021
6. wyfo

   closed this as completedon Oct 10, 2021
7. Fidget-Spinner commented on Oct 10, 2021

   

   Fidget-Spinner

   on Oct 10, 2021

   Member

   More actions

   No worries!

   So I'll have to add some if in my code.

   Yeah, we had to do that in the typing module too. Hope you manage to fix your library without much trouble.
8. gvanrossum commented on Oct 10, 2021

   

   gvanrossum

   on Oct 10, 2021

   Member

   More actions

   If you meant to say: why is typing.Union[] allowed, but not
   types.UnionType[]? That is intentional. types.UnionType is only meant for
   builtin types. Once you union with *any* type from typing, it will convert
   to a typing.Union.

   But why? Just so types.UnionType (if it has a typevar) doesn’t have to
   support subscriptions? Even if this saves us now, I agree with OP that it
   ought to allow it, so we can deprecate typing.Union properly. And e.g.
   dict[str, T] works.

   --Guido (mobile)
9. Fidget-Spinner commented on Oct 10, 2021

   

   Fidget-Spinner

   on Oct 10, 2021

   Member

   More actions

   @guido,

   I hope I didn't misunderstand you, but to clarify, what OP is asking is an alternative way to construct types.UnionType objects and write:

   types.UnionType[int, str]

   like how we used to write before 3.10:

   typing.Union[int, str]

   I don't know why we need this. We can write int | str. The reason for PEP-604 in the first place was to avoid the subscript syntax and use | since it's cleaner. OP's use case is for reconstructing types.UnionType objects easily, but functools.reduce(operator.or_, args) works.

   Re: TypeVar subscription; PEP-604 syntax already supports that. We used to implement that in C. After Serhiy's Great Cleanup, a bitwise OR with a TypeVar automatically converts types.UnionType to typing.Union. So all the TypeVar support is now done in Python.

   >>> type(int | str)
   <class 'types.UnionType'>
   
   >>> (int | str | T)[dict]
   typing.Union[int, str, dict]
10. gvanrossum commented on Oct 10, 2021

    

    gvanrossum

    on Oct 10, 2021

    Member

    More actions

    Oh, I see. No, they must make another special case, like for Annotated.--
    --Guido (mobile)
11. serhiy-storchaka commented on Oct 10, 2021

    

    serhiy-storchaka

    on Oct 10, 2021

    Member

    More actions

    types.UnionType corresponds to typing._UnionGenericAlias, not typing.Union.

    We can make (int | str | T)[dict] returning an instance of types.UnionType instead of an instance of typing._UnionGenericAlias. But it will be a breaking change, because typing._UnionGenericAlias and types.UnionType are different and not completely compatible types. We should wait some time before making such changes, so all user code will be made supporting both typing._UnionGenericAlias and types.UnionType.

    If the user code does something special like substituting int | Collection[int] to int | list[int], it should have some additional ifs in any case, otherwise it will not recognize new typing types including types.UnionTypes. And subscription does not work in all typing types, we have copy_with() for some types and special cases for others in the code of the typing module. I am going to unify it finally, but it takes time, my time and user's time to migrate to new idioms.
12. ezio-melotti

    transferred this issue fromon Apr 10, 2022