Preserve original representation for integers / floats in docstrings

@warsaw

BPO	16801
Nosy	@warsaw, @birkenfeld, @rhettinger, @terryjreedy, @larryhastings, @ezio-melotti, @merwok, @bitdancer, @serhiy-storchaka, @pablogsal, @isidentical

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2012-12-28.13:04:13.871>
labels = ['interpreter-core', 'type-feature', '3.11']
title = 'Preserve original representation for integers / floats in docstrings'
updated_at = <Date 2021-08-20.10:10:09.486>
user = 'https://github.com/larryhastings'

bugs.python.org fields:

activity = <Date 2021-08-20.10:10:09.486>
actor = 'mark.dickinson'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2012-12-28.13:04:13.871>
creator = 'larry'
dependencies = []
files = []
hgrepos = []
issue_num = 16801
keywords = []
message_count = 35.0
messages = ['178383', '178384', '178385', '178392', '178397', '178398', '178457', '178470', '178474', '178487', '178488', '178490', '178510', '178519', '178521', '178542', '178647', '178663', '178843', '178856', '178931', '181497', '181504', '182754', '182816', '182862', '182863', '182908', '182910', '182921', '381023', '396721', '396728', '396745', '396746']
nosy_count = 14.0
nosy_names = ['barry', 'georg.brandl', 'rhettinger', 'terry.reedy', 'larry', 'ezio.melotti', 'eric.araujo', 'Arfrever', 'r.david.murray', 'carsten.klein@axn-software.de', 'serhiy.storchaka', 'pablogsal', 'BTaskaya', 'Scott Stevens']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue16801'
versions = ['Python 3.11']

The line declaring the function dbm.open looks like this:
def open(file, flag='r', mode=0o666):

The docstring for dbm.open looks like this:
open(file, flag='r', mode=438)

Obviously 438==0o666. But the author used the octal representation because it's more readable. Unfortunately Python throws that enhanced readability away when it round-trips the rvalue from a string into an integer and back into a string again for the docstring.

It might be an improvement if Python preserved the original source code's representation for integer (and perhaps float) default arguments for parameters. I haven't looked at the code that does the parsing / builds the docstring, but I suspect we could hang the original representation on the AST node and retrieve it when building the docstring.

The only problem I can forsee: what about code that uses local variables, or computation including perhaps function calls, to calculate default values? On the one hand, the local variable or the function call may be inscrutable--on the other, perhaps the magic integer value it replaced was no better. Or we could have a heuristic, like if the original representation contains internal spaces or parentheses we use str(rvalue), otherwise we use the original representation.

(I was also considering proposing using annotations to tell the parser we want the original representation in the docstring, but I suspect that's a bad idea. That would instantly restrict the untamed frontier that is annotations.)

It's an interesting idea. This sounds like the wrong solution to me, though: it's significant extra machinery to produce a solution that only fixes a small handful of cases; IOW, the benefit / cost ratio seems to small to make this worth it. E.g., apart from the function calls that you mention, what about expressions? "-0x8000" isn't a numeric literal, so the 'original representation' information attached to "0x8000" will have been lost.

I'm also sceptical that this can be done as simply as you describe: isn't the AST no longer available at the time that the docstring is built?

Perhaps what we need instead is a general mechanism to override the generated signature line?

I don't think you mean 'docstring'. A docstring is something a human writes in the source code. I presume you are actually talking about introspection of the signature here. Beyond that, I agree with Mark's comments.

BTW, in case it saves anyone else some time, the current machinery is in Lib/pydoc.py, in the TextDoc.docroutine method. It uses inspect.getfullargspec to retrieve the information to format, though I guess using inspect.Signature would be the modern way to do this.

Bah. s/inspect.Signature/inspect.signature/

David is correct

>>> dbm.open.__doc__
"Open or create database at path given by *file*.\n\n    Optional argument *flag* can be 'r' (default) for read-only access, 'w'\n    for read-write access of an existing database, 'c' for read-write access\n    to a new or existing database, and 'n' for read-write access to a new\n    database.\n\n    Note: 'r' and 'w' fail if the database doesn't exist; 'c' creates it\n    only if it doesn't exist; and 'n' always creates a new database.\n    "
>>> help(dbm.open)
Help on function open in module dbm:

open(file, flag='r', mode=438)
    Open or create database at path given by *file*.
...

IDLE tooltip (still using inspect.getfullargspec) also shows
open(file, flag='r', mode=438)
The int comes from dbm.open.__defaults__[1]
438

Okay, counter-proposal time. We add a new field to the Parameter object, the preferred string representation of the default. If the parameter has a default, it is always a string, by default repr(parameter_default_value); if the parameter has no default then it is None.

You can then override the default:

@inspect.use_original_representation('mode')
def open(file, flag='r', mode=0o666):

And if os.open were supplied in os.py:

@inspect.override_string_representation('mode',
    'os.O_CREAT | os.O_RDWR')
def open(file, flags, mode=0o777, *, dir_fd=None):

(p.s. I know the mode argument there is wrong, it has a million things in it and this is just to get the idea across.)

Then the docstring generator, IDLE, etc. would be told "Please use this new field of the Parameter object when displaying the default to the user."

If you wish to pursue this, I suggest starting with 'the simplest thing that works' for the text cases at hand. They all involve 'mode' and you have not presented and I cannot think of other cases. So somewhere in the signature generation code:
if function name in <set> and parameter_name == 'mode':
replace_decimal_with_octal(parameter_default)

If more generality is really needed, pick a new reserved attribute for functions and set it at the time of definition.
def open(file, flag='r', mode=0o666):
open.__param_rep__ = {'mode': 'octal'} #or whatever is chosen

I suppose the advantage of adding the syntactic sugar of a decorator, after getting the above to work, is that the doc could be hidden away in the inspect model, where is would be easily ignored.\

Still, this does seem like a lot of 'noise' for a small bit of extra 'signal' increment.

The problem with this is that at the time that pydoc gets the information via inspect, the numbers have already been parsed as long or double and the original notation is no longer available.

This is due to the fact that during build of the AST node for the NUMBER type, the value will already be deserialized into its machine representation, which is either long or double.

The only way to preserve that information would be to extend the NUM_type with an additional 's' field which then would preserve its original notation and which can be retrieved from the AST.

pydoc, however, would still fail as it does not use the AST. In order to restore the original information, pydoc must then source the original file or source of the function or class method and parse it using the AST.

A much simpler approach would be to simply get the function or method source and extract its formal parameter list using for example a regular expression.

However, preserving the original notation in the runtime is not required and shouldn't be done.

Here are some links into the sources:

Python/ast.c, ast_for_atom(), line 1872ff.
Python/ast.c, parsenumber(), line 3632ff.

However, hinting inspect to use a different format when serializing the default values for existing keyword parameters of methods or functions
seems to be a good idea and +1 by me for that.

Personally, I'd rather have the decorator based solution than having to manually add additional fields to a given method or function.

And if os.open were supplied in os.py:

@inspect.override_string_representation('mode',
    'os.O_CREAT | os.O_RDWR')
def open(file, flags, mode=0o777, \*, dir_fd=None):

Other use case is a sentinel default. "foo(arg={})" looks better than "foo(arg=<object object>)" for function which use a sentinel idiom:

    _sentinel = object()
    def foo(arg=_sentinel):
        if arg is _sentinel:
            arg = {}
        ...

Sometimes full signature overwriting is needed (see for example Python implementation of operator.methodcaller() in bpo-16694).

Please open a ticket! This one is about docstrings and pydoc; docs.python.org is built from rst docs with sphinx.

I suspect the difference is due to a change in the way that Sphinx handles the py:method directive: the rst source for pathlib.Path.touch hasn't changed between 3.9 and 3.10, but the 3.9 docs are built with Sphinx 2.4.4, while the 3.10 docs are built with Sphinx 3.2.1.

FWIW, the relevant change seems to be this one: sphinx-doc/sphinx#7155

This topic came up again in a recent chat about the sys.maxsize default value being replaced with a large cryptic integer value when reporting the signature of list.index.

Since the early iterations of the discussion, enum.IntEnum also entered the standard library, allowing many cases of magic numbers to be given a representation as a set of named values.

int subclasses with different default representations (octal/hex/binary) still seem like the cleanest approach to handling cases like reporting default file permissions, as it is impossible to infer the intended semantics from the values themselves.

What, if anything, could reasonably be done to improve the handling of values like sys.maxsize, sys.maxunicode, math.e, math.pi, or math.tau when they appear as default values in function signatures is markedly less clear.

First, I think that initial idea seems to be working. At least for pure-Python code - we can get original default value:

>>> import ast, dbm, inspect
>>> s = inspect.getsource(dbm.open)
>>> t = ast.parse(s)
>>> t.body[0].args.defaults[1].lineno
1
>>> t.body[0].args.defaults[1].col_offset
30
>>> t.body[0].args.defaults[1].end_col_offset
35
>>> s.splitlines()[1-1][30:35]  # that damn value
'0o666'

That works also for algebraic expressions:

>>> import ast, inspect
>>> def foo(x=2**5 - 1):
...     pass
...     
>>> s = inspect.getsource(foo)
>>> t = ast.parse(s)
>>> t.body[0].args.defaults[0].lineno
1
>>> t.body[0].args.defaults[0].col_offset
10
>>> t.body[0].args.defaults[0].end_col_offset
18
>>> s.splitlines()[1-1][10:18]
'2**5 - 1'

So, it seems we indeed can select a custom (preferred by programmer in source code) expression for the default value, preparing the Signature object for a callable. Obviously, that may work also for C extensions, providing the __text_signature__ dunder: we can construct "fake source code" for them (i.e. just definitions for functions - no body).

Is this worth efforts? I'm not sure. It's looking as a niche solution, while problem is more generic. Will 2**63-1 be more descriptive than 9223372036854775807?

I also think that some mixin class, making a custom repr - might be a better approach here. Together with the repr - it can customize also docstring: I think it will be more helpful if we show something better for math.pi.__doc__, instead of the float class docs. (Here an example of related issue: #104042)

Cryptic literal values, instead of some symbolic - prevent from using them as defaults. For instance, we can't use math.e as default for base argument of the math.log (related issue: #89381):

>>> import inspect, math
>>> def mylog(x, base=math.e, /):
...     return math.log(x, base)
...     
>>> inspect.signature(mylog)
<Signature (x, base=2.718281828459045, /)>

With a custom named floating-point constant it's possible:

>>> import inspect, math
>>> math.e
math.e
>>> math.e.__doc__
'Euler number (natural logarithm base).'
>>> type(math.e)
<class 'collections._NamedFloat'>
>>> isinstance(math.e, float)
True
>>> inspect.signature(math.log)
<Signature (x, base=math.e, /)>

(Draft patch is available to play: skirpichev#22)

Edit: I've opened a d.p.o thread https://discuss.python.org/t/107540

pythongh-61005: preserve original source for default in repr(Parameter)

Alternative approach (close to original proposal): skirpichev#23

Obviously, it's a breaking change. The pr still have some broken tests, that rely on evaluated default arguments in the repr. Let me know if it worth to open a real pr.

I think that current formatting of the Signature's is not helpful and it's essentially prevents from using complex defaults in the API. Something like sys.maxsize or math.pi in repr for defaults would be much more descriptive.

mark _thread.interrupt_main signature unsupported

Navigation Menu

Repository navigation

Preserve original representation for integers / floats in docstrings #61005

19 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Preserve original representation for integers / floats in docstrings #61005

Description

Activity

larryhastings commented on Dec 28, 2012

larryhastings commented on Dec 28, 2012

mdickinson commented on Dec 28, 2012

bitdancer commented on Dec 28, 2012

mdickinson commented on Dec 28, 2012

mdickinson commented on Dec 28, 2012

terryjreedy commented on Dec 29, 2012

larryhastings commented on Dec 29, 2012

terryjreedy commented on Dec 29, 2012

carstenkleinaxn-softwarede commented on Dec 29, 2012

carstenkleinaxn-softwarede commented on Dec 29, 2012

carstenkleinaxn-softwarede commented on Dec 29, 2012

serhiy-storchaka commented on Dec 29, 2012

19 remaining items

merwok commented on Jun 29, 2021

mdickinson commented on Jun 29, 2021

mdickinson commented on Jun 29, 2021

ncoghlan commented on Jul 11, 2024

skirpichev commented on May 27, 2026

skirpichev commented on Jun 26, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions