Repository navigation
Proposal: Convert the typing-module PEPs into a series of HOWTOs #91533
Description
Problem description
The typing module docs have a problem.
typing.rst is littered with references to the many typing-related PEPs that have been accepted by the Steering Council. Often these references to PEPs are used in lieu of detailed examples or explanation as to how a feature is supposed to work.
However, PEPs are not supposed to be living pieces of documentation. As such, proposals to update old typing PEPs to reflect changes in implementation details have been rejected. But when these PEPs are the closest thing we have to a full Python-typing specification, this means that our specifications are slowly growing out of date.
This unfortunate situation has also been recognised by the Steering Council in a recent email to the typing-sig mailing list.
Proposed solution
I propose that we slowly start to convert various typing PEPs into a series of Python-typing HOWTOs. In this way, we'd follow the example of other modules such as enum or socket. These modules have terse API references in socket.rst and enum.rst, but these API references are supplemented with expansive HOWTOs that serve as a guide for users who are new to the respective modules. Given the number of typing PEPs, the scale of the task for typing will be larger; but I still believe that this is a good example to follow.
Specifically, I think we should have something like the following structure:
- "Introduction to type hints in Python". This HOWTO would combine material from the following sources (updating examples where relevant to use modern syntax such as PEP-604 union types).
- PEPs 483 & 484 (introducing type hints to Python)
- PEP 526 (syntax for variable annotations,
ClassVar) - PEP 613 (explicit type aliases)
- PEP 673 (
typing.Self) - The sections in
typing.rstonAny,Callableand type aliases. - Some of the material in
typing.rstonTypeVarand generics (but only the most basic material).
- "Generics HOWTO". While the intro HOWTO would introduce the basic concept of TypeVars, this HOWTO would tackle the nitty-gritty:
- Using
typing.Genericto make generic classes. - The differences between bound TypeVars, constrained TypeVars, and TypeVars which are neither.
- Covariance versus contravariance versus invariance.
- Maybe a brief mention of PEP 585 (generic stdlib collections).
ParamSpecandConcatenate(material from PEP 612).- Variadic generics (material from PEP 646).
- Using
- "Structural subtyping HOWTO", using material from PEP 544 and the section in
typing.rston structural subtyping andtyping.Protocol. - "TypedDict HOWTO", using material from PEPs 589 & 655, as well as the section on
TypedDictintyping.rst. - "Literal types HOWTO", describing how to use PEPs 586 (
typing.Literal) & 675 (typing.LiteralString).
Due to the brevity of their PEPs, and the fact that they don't really "fit in" neatly with any of the other proposed HOWTOs, I propose that the following typing features have their full specifications integrated into the typing.rst API reference, rather than having dedicated HOWTOs:
- "
Final/@final(PEP 581) Annotated(PEP 593)TypeGuard(PEP 647)
Why not have this information at typing.readthedocs.io?
There's an ongoing effort, spearheaded by @srittau among others, to include a body of typing documentation at https://typing.readthedocs.io/en/latest/. I am fully in support of this effort, and see the documentation there as complementary to the documentation that currently lives in the typing PEPs. However, I do not believe that the documentation currently in the typing PEPs should belong on that site, for several reasons:
- https://typing.readthedocs.io/en/latest/ is a third-party site, rather than being an "official" corpus of Python-language documentation. This is a good thing -- it allows contributors to that site to iterate and improve on the documentation there at a more rapid pace. However, the information in the
typingPEPs are officially recognised changes to the language and the language's typing framework. As such, -- in my opinion -- it should live in the central docs.python.org documentation along with the documentation for the rest of the language. - https://typing.readthedocs.io/en/latest/ is currently not particularly discoverable via search engines. For example, it does not appear in the first page of Google results for "Python typing".
Questions to be discussed
- What do we think of the general idea of moving material out of PEPs, providing specifications through a series of HOWTOs, and slimming down
typing.rstinto a terser API reference? - What do we think of my specific proposal for how to structure this series of HOWTOs?
Copied to:
typingmodule maintainers: @gvanrossum, @Fidget-Spinner, @JelleZijlstra- @srittau, who does a lot of valuable work on typing documentation over at the https://github.com/python/typing repo.
- @hauntsaninja, who recently expressed interest in helping with typing docs.
- Several miscellaneous typing-PEP authors who may be interested in this proposal: @ambv, @JukkaL, @ilevkivskyi, @mrkmndz, @pradeep90, @Gobot1234, @mrahtz, @davidfstr, @erictraut
- @encukou, who recently sent the email to typing-sig on this topic on behalf of the Steering Council, and is also a member of the docs working group.
Activity
[-]Convert the `typing`-module PEPs into a series of HOWTOs[/-][+]Proposal: Convert the `typing`-module PEPs into a series of HOWTOs[/+]AlexWaygood commentedon Apr 14, 2022
(In case it's not obvious: I'm happy to invest time and energy into writing some of these, if we agree that this is a good solution to the problem.)
srittau commentedon Apr 14, 2022
I would invite you to have this discussion over at https://github.com/python/typing/issues. The documentation over there (which is currently published to https://typing.readthedocs.io) is intended to be the comprehensive source for typing related documentation. There already are a few issues about a possible structure on that site. Fragmenting the existing documentation even more should not be the goal.
readthedocs was chosen for an easy to set-up documentation target. Eventually integrating the documentation with docs.python.org should be a goal, though.
srittau commentedon Apr 14, 2022
Also cc @shannonzhu who set up a lot of the infrastructure of the typing docs.
AlexWaygood commentedon Apr 14, 2022
The core issue, in my opinion, is that the
typingdocs at https://docs.python.org/3/library/typing.html currently link far too much totypingPEPs. These are not meant to be living pieces of documentation, and so are rapidly becoming outdated. This is an issue with thetypingdocs living in the CPython repo, so I think it makes sense to have this conversation in the CPython issue tracker.One possible alternative solution to the problem, as you say, is to invest energy into further expanding the documentation at https://typing.readthedocs.io/, and link heavily to that corpus of documentation from the docs at https://docs.python.org/3/library/typing.html.
encukou commentedon Apr 14, 2022
This sounds great to me!
One argument for typing.readthedocs.io is that a lot of the information is not tied to Python versions, or is usable from third-party backport modules.
It's fine to mention typing-extensions in Python docs, but keeping the docs updated across Python versions might be too much of a hassle. And typing.readthedocs.io could have things like proposed extensions, tool compatibility overviews.
IMO, typing.readthedocs.io can be made official enough by linking to it from docs.python.org.
Maybe a good process would be to draft articles on RTD and move to HOWTOs when finalized?
(I'm just throwing ideas around, not arguing a position. Please consider them, but you don't need to convince me.)
JelleZijlstra commentedon Apr 14, 2022
Thanks for starting this effort! I'd been thinking about working on this area too. In my mind, there are several different goals we could work towards:
These two perhaps shouldn't be the same document. A spec is useful for advanced users, but for most users it's not the right place to start. I'd want to focus on the spec, for a couple of reasons: it's easier since we can mostly glue together the spec sections of the PEPs; it allows us to change the spec in the future (which we sometimes want to do); and there are already excellent documentation resources online (e.g., the mypy docs).
So here's what I'd want to see:
typingmodule documentation should focus on listing the names available in the typing module and giving brief introductions to each.I don't feel too strongly about where to put the spec or docs. typing.readthedocs.io is nice but we haven't made much progress with it and it's true that it may be less discoverable.
AlexWaygood commentedon Apr 14, 2022
I'll cross-reference the related issue over at the python/typing repo, so that people can have a more full consideration of the (slightly competing, but I think/hope also somewhat complementary 🙂) proposals:
AlexWaygood commentedon Apr 14, 2022
+1 for looking to the mypy docs as an example -- they're a really impressive body of typing documentation. I'd be wary of linking to them too heavily, though, as they are also becoming outdated in places -- we'd effectively be relying on a third-party project having well-maintained documentation. mypy also can't document any typing features that they don't yet support, whereas CPython should document the latest typing features, even if they are not yet supported by all type-checkers (and full support of new typing features by third-party type checkers can often take >1 year).
gvanrossum commentedon Apr 14, 2022
Would it make sense to kick-start the typing docs for end users by selectively copying sections from the mypy docs (with some kind of attribution)? Just like starting the specs with the existing PEPs.
AlexWaygood commentedon Apr 14, 2022
I guess I'm not 100% sold on the distinction between specifications and end-user documentation. I get the idea, but I worry that we'll end up with duplication between the different pieces of documentation. For example, PEP 544 is mostly a specification. But I've also seen it remarked upon in several places (e.g. here, or here) that it serves as an excellent piece of end-user documentation, which is better than anything currently at https://docs.python.org/3/library/typing.html.
davidfstr commentedon Apr 14, 2022
I agree that it would be useful to collect a set of official “typing Guides/HOWTOs”, which are distinct from the typing-related PEPs (which are deltas applied on top of each other).
I agree that the mypy documentation would be a good starting point for this effort. This is the documentation that I currently consult first for guide-level typing documentation.
I don’t particularly care where such guide level documentation lives. I do think, however that it might be best suited to be in a site external from the versioned Python documentation, since type annotations are mostly agnostic to whatever Python version is in use. (In particular you can use very new typing features in older versions of Python.) I do think, however that typing.rst from the Python documentation should include prominent links to this external site (at least to the site’s top-level) to make it discoverable (both by users and search engines).
I am aware of the existing https://typing.readthedocs.io/en/latest/ site. A cursory reading of that site’s current content feels it is currently used to house detailed typing related documentation that didn’t already make it into a PEP or somewhere else (like the mypy documentation). For example it details the handling of
__all__across current-generation type checkers. In particular I do not currently see any entry-level guides like “Introduction to optional static typing in Python” or “Why should I care about type annotations in Python?” on that site at this time.It would make sense to me to focus this documentation effort on improving the content at the existing https://typing.readthedocs.io/en/latest/ site, and linking it to from the official versioned Python documentation when the site is deemed “ready” (for some definition of “ready”).
Also, I like the organization of guides proposed by the originator of this thread. ( @AlexWaygood )
AlexWaygood commentedon Apr 14, 2022
It sounds like the prevailing mood is to invest energy into improving the corpus of documentation at typing.readhedocs.io, and then — once they're ready — link heavily from
typing.rstin the CPython docs to that body of documentation. Others in this thread have made persuasive arguments as to why that might be preferable, so I'll happily concede that point.I still have some concerns about the discoverability of typing.readhedocs.io. Does anybody have any idea why it fares so poorly in Google results? Is this just due to the fact that it's a fairly new site, with few external sites linking to it? How can we improve the situation?
I hate the idea of investing large amounts of energy into writing typing docs, only for no one to see them.
7 remaining items
adam-grant-hendry commentedon Sep 17, 2022
Please please please please please!! I am SO in need of this! (I can't possibly be the only newcomer who feels the same way)
adam-grant-hendry commentedon Sep 18, 2022
@AlexWaygood I would love to add this to use cases:
*argsas a Type Variable Tuple?I would also love if another link to the
typeshedreadthedocs could be added in an appropriate place:adam-grant-hendry commentedon Sep 18, 2022
If for nothing else, I think you would help lift the burden on yourselves of having to handle multiple incorrect/false/erroneous issues popping up on the
mypyissue tracker (or intyping).The longer the issues/discussions lists get, the harder it becomes for new users to search through them to find a related issue/discussion, which only serves to add fuel to the fire.
From a 30,000-foot view, this certainly feels like it would be a net win for both sides.
adam-grant-hendry commentedon Sep 18, 2022
After re-reading
Why not have this information at typing.readthedocs.io?, this last one might not be relevant...adam-grant-hendry commentedon Sep 18, 2022
Actually, us new users are saying we don't know where to look:
If we don't know where to go, we can't come...
Providing a new user perspective, what I'm lacking is HOW-TO design principles. These arguably come with experience, but this knowledge is beginning to be needed more and more by new users:
If a "business need" is what's required, I feel fairly safe in claiming you could probably grep the issue tracker and discussions for "that's working as intended" compared with actual bugs and find the percentage of the former greatly outweighs the latter.
Regardless of all of that, I think you'll find new users do want to learn the right way of doing things. I'm certainly in that camp. I want to know how to type hint properly and I want to be able to point others to a proper reference so I don't spread misinformation.
However a HOW-TO can be achieved, I am extremely in favor of it.
pythongh-96917: link to typing.readthedocs.io from typing.rst
pythongh-96917: link to typing.readthedocs.io from typing.rst (python…
gh-96917: link to typing.readthedocs.io from typing.rst (#96921)
pythongh-96917: link to typing.readthedocs.io from typing.rst (python…
[3.10] pythongh-96917: link to typing.readthedocs.io from typing.rst (p…
[3.10] pythongh-96917: link to typing.readthedocs.io from typing.rst (p…
gh-96917: link to typing.readthedocs.io from typing.rst (GH-96921)
[3.10] gh-96917: link to typing.readthedocs.io from typing.rst (GH-96921
gh-96917: link to typing.readthedocs.io from typing.rst (GH-96921)
pythongh-91533: Move much of `typing.rst` to a new 'Guide to typing f…