[DOC] typing.get_type_hints() raises for type aliases with forward references

@gvanrossum

BPO	44926
Nosy	@gvanrossum, @ambv, @mhils, @miss-islington, @Fidget-Spinner
PRs	bpo-44926: `get_type_hints`: Add note about type aliases with forward refs #27859 [3.10] bpo-44926: `get_type_hints`: Add note about type aliases with forward refs (GH-27859) #27860

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-10-18.07:55:48.195>
created_at = <Date 2021-08-16.17:15:16.657>
labels = ['3.11', 'type-bug', 'library', '3.10', 'docs']
title = '[DOC] typing.get_type_hints() raises for type aliases with forward references'
updated_at = <Date 2021-10-18.07:55:48.191>
user = 'https://github.com/mhils'

bugs.python.org fields:

activity = <Date 2021-10-18.07:55:48.191>
actor = 'iritkatriel'
assignee = 'docs@python'
closed = True
closed_date = <Date 2021-10-18.07:55:48.195>
closer = 'iritkatriel'
components = ['Documentation', 'Library (Lib)']
creation = <Date 2021-08-16.17:15:16.657>
creator = 'mhils'
dependencies = []
files = []
hgrepos = []
issue_num = 44926
keywords = ['patch']
message_count = 13.0
messages = ['399660', '399662', '399664', '399672', '399675', '399677', '399746', '399774', '399829', '399850', '399852', '399974', '400086']
nosy_count = 6.0
nosy_names = ['gvanrossum', 'docs@python', 'lukasz.langa', 'mhils', 'miss-islington', 'kj']
pr_nums = ['27859', '27860']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue44926'
versions = ['Python 3.10', 'Python 3.11']

Someone reported this rather interesting issue where typing.get_type_hints crashes on type aliases with forward references. The original report is at mitmproxy/pdoc#290. Here's an extended minimal example:

foo.py:

import typing

FooList1: typing.TypeAlias = list["Foo"]
FooList2: typing.TypeAlias = typing.List["Foo"]

class Foo:
    pass

bar.py:

import typing
import foo

def func1(x: foo.FooList1):
    pass

def func2(x: foo.FooList2):
    pass

print(typing.get_type_hints(func1))  # {'x': list['Foo']}
print(typing.get_type_hints(func2))  # NameError: name 'Foo' is not defined.

Observations:

func1 doesn't crash, but also doesn't resolve the forward reference. I am not sure if this expected behavior.
If it isn't, this should eventually run in the same problem as func2.
func2 crashes because "Foo" is evaluated in the context of bar.py (where class Foo is unknown) and not in the context of foo.py. ForwardRef._evaluate would
somehow need to know in which context it was defined. bpo-41249 (TypedDict inheritance doesn't work with get_type_hints) introduced
ForwardRef.__forward_module__, which would be a logical place for that information. I'm not sure if it is a good idea
to use __forward_module__ more widely.
This may end up as quite a bit of complexity for an edge case, I'm fine if it is considered wontfix.
The reason I'm bringing it up is that PEP-613 (Explicit Type Aliases) decidedly allows forward references in type aliases.

For the record, PEP-563 (postponed evaluations) does not change the outcome here. However, postponed evaluations often make it possible to avoid the forward references by declaring the aliases last.

func1 doesn't crash, but also doesn't resolve the forward reference. I am not sure if this expected behavior.

Good observation! That's indeed the current behavior. The reason is a little subtle - list[...] is using a special builtin type (https://docs.python.org/3/library/stdtypes.html#types-genericalias), while typing.List is the one from typing. The typing version wraps all strings in ForwardRef(), while the builtin version shouldn't import from typing, so it doesn't have that luxury. get_type_hints could probably support the builtin version, though I'd imagine we'd need some rework.

Re: 2. You're right there too. Storing the defined module in __forward_module__ is an interesting proposal.

For your specific use case (where the user is using Python 3.6), you could pass in globalns and localns to get_type_hints as a temporary workaround. Off the top of my head:
get_type_hints(func2, globalns=foo.__dict__) might work. Would that work for your library?

For your specific use case (where the user is using Python 3.6), you could pass in globalns and localns to get_type_hints as a temporary workaround. Off the top of my head:
get_type_hints(func2, globalns=foo.__dict__) might work. Would that work for your library?

I guess the hard part is knowing that the type annotation comes from foo. In the example here we can of course hardcode it, but that doesn't work in the general case or for pdoc, the documentation generator I'm working on (https://pdoc.dev). I have experimented quite a bit with walking the AST to figure out where type aliases are imported from to then re-executing ForwardRefs with that globalns. Long story short, trying to reverse-engineer forward_module quickly becomes a tangled hot mess where you need to adjust for import aliases, reimports, and so on.

@mhils

I just got bit by this, and on a case where eliminating the forward references isn't possible, even with delayed evaluation:

JsonScalar: TypeAlias = str | int | float | bool | None
JsonArray: TypeAlias = list["JsonValue"]
JsonObject: TypeAlias = dict[str, "JsonValue"]
JsonCollection: TypeAlias = JsonArray | JsonObject
JsonValue: TypeAlias = JsonScalar | JsonCollection

I thought of a radical idea for how to fix this: have string literals track the context in which they are defined. So, to use @mhils's example, get_type_hints() would know that func1()'s 'x' argument is a foo.Data and not a bar.Data because the string literal which is stored in foo.FooData knows that it was defined in the foo module at the module level.

Now, the obvious issues are that this would require adding low-level support for a very specific use-case to a very fundamental part of the language, and the potential for performance penalties. For the latter, I note that this extra source information would only need to be tracked for string literals, not all strings, and the number of string literals in any code-base is inherently finite and usually relatively small.

For the former, well, it just comes down to how much being able to reliably evaluate types at runtime is. I, for one, frequently deal with cases where it would be helpful, such as converting structured data (particularly dataclasses) to and from JSON.

Navigation Menu

Repository navigation

[DOC] typing.get_type_hints() raises for type aliases with forward references #89089

17 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[DOC] typing.get_type_hints() raises for type aliases with forward references #89089

Description

Activity

mhils commented on Aug 16, 2021

Fidget-Spinner commented on Aug 16, 2021

mhils commented on Aug 16, 2021

17 remaining items

macdjord commented on Jul 5, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions