__len__ can't return big numbers #65643

Activity

1. cool-RR commented on May 5, 2014

   

   cool-RRmannequin

   on May 5, 2014

   MannequinAuthor

   More actions

   I want to use big numbers for length.

   >>> class A:
   ...     __len__ = lambda self: 10 ** 20
   >>> len(A())
   Traceback (most recent call last):
     File "<pyshell#5>", line 1, in <module>
       len(A())
   OverflowError: cannot fit 'int' into an index-sized integer
2. cool-RR

   added

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   on May 5, 2014
3. bitdancer commented on May 6, 2014

   

   bitdancer

   on May 6, 2014

   Member

   More actions

   While this is classed as a CPython implementation detail (see bpo-15718) it doesn't sound like it is likely to be changed (see bpo-2723).
4. mdickinson

   changed the title [-]__len__ can't return big numbers[/-] [+]__len__[/+] on May 6, 2014
5. mdickinson

   changed the title [-]__len__[/-] [+]__len__ can't return big numbers[/+] on May 6, 2014
6. mdickinson commented on May 6, 2014

   

   mdickinson

   on May 6, 2014

   Member

   More actions

   Whoops; sorry -- accidental title change by typing __len__ into something that wasn't the search box.

   Stupid fingers...

   (I suspect this issue is a duplicate of an existing issue.)
7. bitdancer commented on May 6, 2014

   

   bitdancer

   on May 6, 2014

   Member

   More actions

   Mark: I thought it was too, but the two I noted were the closest I could find. Maybe you'll find something even more on point :)
8. 4kir4 commented on May 6, 2014

   

   4kir4mannequin

   on May 6, 2014

   Mannequin

   More actions

   If len() signature can't be changed to return Python int objects (unlimited) then the OverflowError may contain the actual .length
   property instead (based on msg66459 by Antoine Pitrou)

   operator.length():

       def length(sized):
           """Return the true (possibly large) length of `sized` object.

       It is equivalent to len(sized) if len doesn't raise 
       OverflowError i.e., if the length is less than sys.maxsize on
       CPython; otherwise return OverflowError.length attribute
       """
       try:
           return len(sized)
       except OverflowError as e:
           return e.length
   
9. mdickinson commented on May 6, 2014

   

   mdickinson

   on May 6, 2014

   Member

   More actions

   That's pretty evil. :-)
10. rhettinger commented on May 8, 2014

    

    rhettinger

    on May 8, 2014

    Contributor

    More actions

    I recommend this be closed: too much impact on existing code for too little benefit.

    CPython has historically imposed some artificial implementation specific details in order make the implementation cleaner and faster internally (i.e. a limit on the number of function arguments, sys.maxsize limits, etc.)
11. rhettinger

    added

    type-featureA feature request or enhancementA feature request or enhancement

    on May 8, 2014
12. benjaminp

    closed this as completedon May 8, 2014
13. ezio-melotti

    transferred this issue fromon Apr 10, 2022
14. rhettinger commented on Apr 29, 2022

    

    rhettinger

    on Apr 29, 2022

    Contributor

    More actions

    If people really care about this (I'm not sure they do), there is a possible general solution.

    1. Add an OverflowError fallback code path to the builtin len() function. That path can then compute arbitrarily large sizes from objects that have integer start, stop, and step attributes. The fixes cases like len(range(1 << 100)).

    2. The builtin len() could also be built out with a fallback that calls __len__ method directly. This won't help with builtin types like rangeobject, but it would help with user defined types:

    class A:
        def __len__(self):
            return 1 << 66
    
    >>> a.__len__()
    73786976294838206464
    

    These two measures would mitigate the limitations imposed by CPython's system of internal slots like mp_length and sq_length. However, I don't think any work should be done until we've ascertained whether we're partially solving a real problem or just addressing a theoretical concern.
15. serhiy-storchaka commented on Apr 30, 2022

    

    serhiy-storchaka

    on Apr 30, 2022

    Member

    More actions

    (1) can only fix len() for range(), (2) only works with Python implementations and may call __len__ twice.

    Adding an attribute to OverflowError and catching it in len() could solve both problem, but it is a bit unsafe, because an OverflowError can be raised in immediate computations.

    The only reliable method is adding slots mp_long_length and sq_long_length. But it is too heavy change for solving the problem for which we don't know how often it occurs in real code.