Should str.format allow negative indexes when used for __getitem__ access?

@rhettinger

BPO	7951
Nosy	@rhettinger, @terryjreedy, @mdickinson, @ericvsmith, @merwok, @florentx, @rovitotv, @marco-buttu, @ikamensh, @iritkatriel
Files	format_negative_indexes-2.7.diff: patch against trunk format_negative_indexes-3.2.diff: patch against 3.2 format_no_fields_with_negative_indexes-2.7.diff: Don't allow negative fields 7951NegativeIndexesForStringFormat3dot4.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = <Date 2021-12-01.01:47:31.771>
created_at = <Date 2010-02-17.23:54:17.722>
labels = ['easy', '3.11', 'type-feature', 'docs']
title = 'Should str.format allow negative indexes when used for __getitem__ access?'
updated_at = <Date 2021-12-01.01:47:31.769>
user = 'https://github.com/ericvsmith'

bugs.python.org fields:

activity = <Date 2021-12-01.01:47:31.769>
actor = 'eric.smith'
assignee = 'terry.reedy'
closed = True
closed_date = <Date 2021-12-01.01:47:31.771>
closer = 'eric.smith'
components = ['Documentation']
creation = <Date 2010-02-17.23:54:17.722>
creator = 'eric.smith'
dependencies = []
files = ['17712', '17713', '17768', '29948']
hgrepos = []
issue_num = 7951
keywords = ['patch', 'easy']
message_count = 33.0
messages = ['99482', '99553', '107766', '107776', '107781', '107792', '107793', '107801', '107811', '107845', '108132', '108133', '108143', '108144', '108472', '108617', '113447', '113620', '113624', '115981', '116244', '187404', '190102', '215958', '216038', '225505', '266481', '340877', '340884', '347795', '407301', '407328', '407422']
nosy_count = 14.0
nosy_names = ['rhettinger', 'terry.reedy', 'mark.dickinson', 'eric.smith', 'kisielk', 'eric.araujo', 'mrabarnett', 'flox', 'docs@python', 'gosella', 'Todd.Rovito', 'marco.buttu', 'Ilya Kamenshchikov', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue7951'
versions = ['Python 3.11']

It surprised me that this doesn't work:
>>> "{0[-1]}".format('fox')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: string indices must be integers

I was expecting it to be equivalent to:

>>> "{0[2]}".format('fox')
'x'

I don't think there's any particular reason this doesn't work. It would, however break the following code:

>>> "{0[-1]}".format({'-1':'foo'})
'foo'

But note that this doesn't work currently:

>>> "{0[1]}".format({'1':'foo'})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 1

On a related note, this doesn't work either:

>>> "{-1}".format("x", "y", "z")
Traceback (most recent call last):
  File "<pyshell#3>", line 1, in <module>
    "{-1}".format("x", "y", "z")
KeyError: '-1'

It could return "z".

It also rejects a leading '+', but that would be optional anyway.

Closed bpo-8985 as a duplicate of this; merging nosy lists.

I (reluctantly) agree it's surprising that "{0[-1]}".format(args) fails. And I suppose that if it were allowed then it would also make sense to consider "{-1}".format(*args) as well, in order to preserve the equivalence between "{n}".format(*args) and "{0[n]}".format(args). And then:

>>> "{-0}".format(*['calvin'], **{'-0': 'hobbes'})
'hobbes'

would presumably produce 'calvin' instead of 'hobbes'...

On '+': if "{0[-1]}" were allowed, I'm not sure whether the "+1" in "{0[+1]}".format(...) should also be interpreted as a list index. I don't really see the value of doing so apart from syntactic consistency: there are very few other places in Python that I'm aware of that accept -<one-or-more-digits> but not +<one-or-more-digits>.

FWIW, my overall feeling is that the current rules are simple and adequate, and there's no great need to add this complication.

I do wonder, though:

How complicated would it be to make "{0[1]}".format({'1':'foo'}) a bit magical? That is, have the format method pass an integer to __getitem__ if the corresponding format argument is a sequence, and a string argument if it's a mapping (not sure what the criterion would be for distinguishing). Is this too much magic? Is it feasible implementation-wise?

I don't think it's do-able for simple rather than compound field names: e.g., "{0}".format(args, **kwargs), since there we've got both a sequence *and a dict, so it's not clear whether to look at args[0] or kwargs['0']. (Unless either args or kwargs is empty, perhaps.) This is all getting a bit python-ideas'y, though.

BTW, I notice that PEP-3101's "Simple field names are either names or numbers [...] if names, they must be valid Python identifiers" isn't actually true:

>>> "{in-valid #identifier}".format(**{'in-valid #identifier': 42})
'42'

Though I don't have a problem with this; indeed, I think this is preferable to checking for a valid identifier.

Addressing just the last part of Mark's message right now:

The PEP goes on to say:
Implementation note: The implementation of this proposal is
not required to enforce the rule about a simple or dotted name
being a valid Python identifier. ...

I rely on getattr lookup failing for dotted names, but for simple names there's no check at all. I agree it's desirable to leave this behavior.

Re: msg107776.

If it looks like an integer (ie, can be converted to an integer by 'int') then it's positional, otherwise it's a key. An optimisation is to perform a quick check upfront to see whether it starts like an integer.

Matthew:

would that include allowing whitespace, then?

>>> int('\t\n+56')
56

That's a good question. :-)

Possibly just an optional sign followed by one or more digits.

Another possibility that occurs to me is for it to default to positional if it looks like an integer, but allow quoting to force it to be a key:

>>> "{0}".format("foo", **{"0": "bar"})
'foo'
>>> "{'0'}".format("foo", **{"0": "bar"})
'bar'

Or is that taking it too far?

I can see the point of allowing negative indices for a consistency point, but is there really any practical problem that's currently causing people hardship that this would solve?

As for the rest of it, I think it's just not worth the additional burden on CPython and other implementations.

Your original:

"{0[-1]}".format('fox')

is a worse gotcha than:

"{-1}".format('fox')

because you're much less likely to want to do the latter.

It's one of those things that it would be nice to have fixed, or we could just add a warning to the documentation that it _might_ be fixed in the future, so people shouldn't rely on the current behaviour. :-)

I finally managed to get the time to finish the patch that allows negative indexes inside square brackets so now they work with the same semantics as in a python expression:

>>> '{0[-1]}'.format(['abc', 'def'])
'def'
>>> '{0[-2]}'.format(['abc', 'def'])
'abc'
>>> '{0[-1][-1]}'.format(['abc', ['def']])
'def'

They work auto-numbered fields too:
>>> '{[-1]}'.format(['abc', 'def'])
'def'

Also, a positive sign is now accepted as part of a valid integer:

>>> '{0[+1]}'.format(['abc', 'def'])
'def'

As a bonus, negatives indexes are also allowed to refer to positional arguments:

>>> '{-1}'.format('abc', 'def')
'def'
>>> '{-2}'.format('abc', 'def')
'abc'

I'm attaching a patch against trunk. I added some tests for this functionality in test_str.py.

By the way, this code doesn't work anymore:

>>> "{[-1]}".format({'-1': 'X'})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: -1L

But now it behaves in the same way as:
>>> "{[1]}".format({'1': 'X'})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 1L

I didn't attempt to ignore whitespaces when trying to parse the index as an integer (to allow that "{ 0 }" can be treated as "{0}" and "{0[1]}" as "{ 0 [ 1 ] }") because I'm not sure if this behavior is desirable.

The doc bug is that the grammar block uses 'integer' (linked to https://docs.python.org/3/reference/lexical_analysis.html#grammar-token-integer) in
  arg_name          ::=  [identifier | integer]
  element_index     ::=  integer | index_string
when it should use 'decimalinteger' or even more exactly 'digit+'. The int() builtin uses the same relaxed rule when no base is given.
>>> 011
SyntaxError: invalid token
>>> int('011')
11
>>> '{[011]}'.format('abcdefghijlmn')
'm'

One possibity is to replace 'integer' in the grammar block with 'digit+' and perhaps leave the text alone. Another is to replace 'integer' with 'index_number', to go with 'index_string, and add the production "index_number ::= digit+". My though for the latter is that 'index_number' would connect better with 'number' as used in the text. A further option would be to actually replace 'number' in the text with 'index_number'.

PS to Todd. As much as possible, doc content changes should be separated from re-formatting. I believe the first block of your patch is purely a re-format

msg216038 suggests three options for the doc patch, does anybody have any preference or a better alternative?

The error message is misleading:

>>> s = '{names[-1]} loves {0[1]}'
>>> s.format(('C', 'Python'), names=('Dennis', 'Guido'))
Traceback (most recent call last):
    ...
TypeError: tuple indices must be integers or slices, not str

A side question: where is it defined that in {thing[0]}, 0 will be parsed as an integer?

The PEP shows {thing[name]} and mentions that this is not Python but a smaller mini-language, with name always a string, no quotes needed or permitted.

I'm not sure where (or if) it's defined in the Python docs, but in PEP-3101 it's in https://www.python.org/dev/peps/pep-3101/#simple-and-compound-field-names: "It should be noted that the use of 'getitem' within a format string is much more limited than its conventional usage. In the above example, the string 'name' really is the literal string 'name', not a variable named 'name'. The rules for parsing an item key are very simple. If it starts with a digit, then it is treated as a number, otherwise it is used as a string.".

Py3.6+ f-strings support any indexing as they actually evaluate python expressions.

>>> a = ['Java', 'Python']
>>> var = f"Hello {a[-1]}"
Hello Python

Reproduced on 3.11, and the error message is a little weirder now:

>>> "{0[-1]}".format('fox')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: string indices must be integers, not 'str'

I recommend not adding support for negative indexing to format() for accessing positional arguments. There is almost no reason to do this because it almost always makes the format string less readable, because the number of arguments is always known in advance, and because the arguments are almost always used entirely rather than selectively.

Negative index support isn't a feature of the language. Instead, it is a feature provided on a class by class basis, if it makes sense for that class and for its use cases.

We are not obliged to provide negative index support in places where it doesn't make sense or where it makes code less readable. For example, the islice() function doesn't support negative indices because it doesn't make sense there. Likewise, the Sequence ABC doesn't require negative index support or slice support.

I'm closing this as "won't fix" for the negative indexing functionality. If someone wants to open an new documentation issue (and ideally provide a PR), that would be welcome.

Navigation Menu

Repository navigation

Should str.format allow negative indexes when used for getitem access? #52199

16 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Should str.format allow negative indexes when used for __getitem__ access? #52199

Description

Activity

ericvsmith commented on Feb 17, 2010

mrabarnett commented on Feb 19, 2010

ericvsmith commented on Jun 13, 2010

mdickinson commented on Jun 14, 2010

ericvsmith commented on Jun 14, 2010

mrabarnett commented on Jun 14, 2010

mdickinson commented on Jun 14, 2010

mrabarnett commented on Jun 14, 2010

ericvsmith commented on Jun 14, 2010

mrabarnett commented on Jun 15, 2010

gosella commented on Jun 18, 2010

16 remaining items

terryjreedy commented on Apr 13, 2014

BreamoreBoy commented on Aug 18, 2014

marco-buttu commented on May 27, 2016

merwok commented on Apr 26, 2019

ericvsmith commented on Apr 26, 2019

ikamensh commented on Jul 13, 2019

iritkatriel commented on Nov 29, 2021

rhettinger commented on Nov 29, 2021

ericvsmith commented on Dec 1, 2021

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Should str.format allow negative indexes when used for getitem access? #52199