Repository navigation
ParamSpec substitution: difference in behaviour between collections.abc.Callable and typing.Callable #102723
Description
There is currently a difference in behaviour between collections.abc.Callable and typing.Callable for the following edge case involving ParamSpec substitution:
Running PGUpdate|x64 interpreter...
Python 3.12.0a6+ (heads/main:12226bec25, Mar 10 2023, 17:32:23) [MSC v.1932 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import typing as t, collections.abc as c
>>> P, T = t.ParamSpec("P"), t.TypeVar("T")
>>> t.Callable[P, T][[P, str], bool][int]
typing.Callable[[int, str], bool]
>>> c.Callable[P, T][[P, str], bool][int]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<frozen _collections_abc>", line 501, in __getitem__
File "<frozen _collections_abc>", line 456, in __new__
TypeError: Callable must be used as Callable[[arg, ...], result].According to PEP-612, this is an invalid substitution, for two reasons:
- Parameters can only be prepended to a
ParamSpecin a parameters list usingtyping(_extensions).Concatenate. - Appending parameters to a
ParamSpecin a parameters list is disallowed.
As such, the behaviour of collections.abc.Callable is more correct here, so ideally we'd change the behaviour of typing.Callable to match collections.abc.Callable.
However, this error should hopefully be caught by static type checkers anyway, and this is a false negative rather than a false positive. The runtime makes no promises that it will raise TypeError on all invalid substitutions, so fixing this should be low priority, in my opinion. We should first concentrate on fixing substitutions where the runtime raises exceptions, even though it shouldn't. For example:
This discrepancy in behaviour was first uncovered as part of a broader discussion in:
Cc. @sobolevn
Activity
sobolevn commentedon Mar 16, 2023
I am going to research, how hard it is to change.
If this is a rather simple patch, I am going to apply it.
If this is a complex one, I will close this issue as "won't fix".
Because parameters and their substitution is a very hard thing to get right.
sobolevn commentedon Mar 16, 2023
Wait, I am confused again. Why
c.Callable[P, T][[P, str], bool]is allowed?It seems like
Pcan only be substituted with:ParamSpecvariable:P->P2...ParamSpecvariables:[int, str]ConcatenatewithParamSpec:Concatenate[int, P]Am I missing something?
AlexWaygood commentedon Mar 16, 2023
You're correct,
c.Callable[P, T][[P, str], bool]should also, strictly speaking, not be allowed.Honestly, I think we should probably leave this one. The runtime allows all sorts of crazy things that should be rejected by type checkers. My personal favourite (we should make this an alias for
Any):>>> list[r"¯\_(ツ)_/¯"] list['¯\\_(ツ)_/¯']We've never promised that the runtime would disallow all things that should be rejected by type checkers, and I don't want us to start making that promise now.
The fact that
c.Callable[P, T][[P, str], bool][int]raises aTypeErroris, strictly speaking, correct -- but I think it's almost certainly just an accident of the current implementation rather than something that was designed. So we shouldn't spend time trying to change the behaviour (since the behaviour is not incorrect), but we also probably shouldn't spend time trying to maketyping.Callablematch it.#88965 seems like a much more important issue to me, since that involves the runtime incorrectly raising a
TypeErrorfor valid substitutions.sobolevn commentedon Mar 16, 2023
Yes, I agree. We should close this as won't fix. Because the correct behaviour is very complex to achieve. And will probably break other things.
There's no point to address a minor issue like this one by the cost of increasing code complexity.
AlexWaygood commentedon Mar 16, 2023
Would you be interested in taking a look at #88965? Serhiy assigned it to himself several months ago, but there's been no activity on the issue since, and it would be great to have it fixed.
sobolevn commentedon Mar 16, 2023
I will tomorrow!