re.finditer and re.findall should support negative end positions

@rhettinger

BPO	7940
Nosy	@rhettinger, @ezio-melotti, @zware, @serhiy-storchaka, @anilbey
PRs	gh-52188: add support for negative end positions to re.finditer and re.findall #14744
Files	issue7940.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2010-02-16.14:04:33.871>
labels = ['type-feature', '3.9', 'docs']
title = 're.finditer and re.findall should support negative end positions'
updated_at = <Date 2019-09-09.16:55:47.901>
user = 'https://bugs.python.org/tlynn'

bugs.python.org fields:

activity = <Date 2019-09-09.16:55:47.901>
actor = 'serhiy.storchaka'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2010-02-16.14:04:33.871>
creator = 'tlynn'
dependencies = []
files = ['30377']
hgrepos = []
issue_num = 7940
keywords = ['patch']
message_count = 15.0
messages = ['99411', '99456', '189882', '189885', '190041', '190101', '190121', '347852', '347913', '347915', '347918', '347958', '347997', '351530', '351534']
nosy_count = 9.0
nosy_names = ['rhettinger', 'tlynn', 'timehorse', 'ezio.melotti', 'mrabarnett', 'docs@python', 'zach.ware', 'serhiy.storchaka', 'anilbey']
pr_nums = ['14744']
priority = 'low'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue7940'
versions = ['Python 3.9']

Linked PRs

gh-52188: emit FutureWarning for negative start/end indices #107105

Python 2.5.2 (r252:60911, Jan 20 2010, 23:14:04) 
[GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import re
>>> list(re.compile('x').finditer('xxxx', 1, 3))
[<_sre.SRE_Match object at 0x7f2e820f09f0>, <_sre.SRE_Match object at 0x7f2e820f0d98>]
>>> list(re.compile('x').finditer('xxxx', 1, -1))
[]
>>> re.compile('x').findall('xxxx', 1, -1)
[]
>>> re.compile('x').findall('xxxx', 1, 3)
['x', 'x']

I discovered that my regex module (see issue bpo-2636) suffers from the same problem. Oops! It was a simple fix (to be released).

Has this been fixed in the regex module?

Yes. As msg99456 suggests, I fixed it the my source code before posting.

Compare re in Python 3.3.2:

>>> re.compile('x').findall('xxxx', 1, 3)
['x', 'x']
>>> re.compile('x').findall('xxxx', 1, -1)
[]

with regex:

>>> regex.compile('x').findall('xxxx', 1, 3)
['x', 'x']
>>> regex.compile('x').findall('xxxx', 1, -1)
['x', 'x']

I've attached a patch.

I'm worrying about backward compatibility. See also bpo-7951.

Like the OP, I would've expected it to handle negative indexes the way that strings do.

In practice, I wouldn't normally provide negative indexes; I'd use some string or regex method to determine the search limits, and then pass them to finditer and findall, so they'd be non-negative anyway.

-1 on the proposal. We don't know of any strong use cases, so there isn't a real problem being solved here. Rather than providing a benefit, this feature request makes it more likely that people will write convoluted code or that it will let bugs pass silently that would otherwise be caught.

ISTM the actual issue here is an incorrect user expectation that "all things that having indexing will support negative indexing". While it is common for objects to implement negative index support, it is not universal or required. Even collections.abc.Sequence does not insist on negative index support.

I think this warrants a FAQ entry (which should also mention that slice support as well is not universal or required, some objects have it, some don't).

Reclassifying this as documentation issue.

The current behavior is inconsistent because the start position accepts both positive and negative indices, whereas the end position only accepts positive indices.
I think the proposal and the PR written by Anil are reasonable and should be merged.

Sorry, I was wrong. re.findall accepts negative indices for both start and end but they silently get converted to 0, which is arguably an unexpected behavior.

This is an example of the current behavior:
>>> s, e = 1, 4; re.compile('.').findall('abcde', s, e), 'abcde'[s:e]
(['b', 'c', 'd'], 'bcd')
>>> s, e = -4, 4; re.compile('.').findall('abcde', s, e), 'abcde'[s:e]
(['a', 'b', 'c', 'd'], 'bcd')
>>> s, e = 1, -1; re.compile('.').findall('abcde', s, e), 'abcde'[s:e]
([], 'bcd')
>>> s, e = -4, -1; re.compile('.').findall('abcde', s, e), 'abcde'[s:e]
([], 'bcd')

With the patch, all these return ['b', 'c', 'd']. This change might indeed cause issues because it's a change in behavior, but I'm also not sure there are many cases where one would want a negative index to be treated as 0. Maybe we could raise a FutureWarning in the next release and change the behavior afterwards?

Are there any real world examples which show the benefit of supporting negative indices?

@mrabarnett

I guess the use of negative indices serve the same purpose here as in lists or strings.
Though as Ezio pointed out, the current behaviour is already accepting negative indices but providing inconsistent results in comparison to various other Python modules that support negative indices.

In my opinion:
If the negative indices are to be used here (which is what the current implementation suggests), they should behave in the same way as the rest of the Python modules.
Otherwise, perhaps negative indices should not be allowed here at all.

What do you think?

P.S. this patch is already applied to the regex module by @mrabarnett

Are there any real world examples which show the benefit of supporting
negative indices?

A common case is ignoring parentheses at the beginning/end, e.g.
>>> re.compile('[^,]+').findall('(foo,123,(),bar)')
['(foo', '123', '()', 'bar)']
>>> # ignore the surrounding ()
>>> re.compile('[^,]+').findall('(foo,123,(),bar)', 1, 15)
['foo', '123', '()', 'bar']
>>>
>>> # extract attributes from a tag (poc, doesn't handle all cases)
>>> re.compile('[^ ]+').findall('<input type="checkbox" id="foo" checked>', 7, 39)
['type="checkbox"', 'id="foo"', 'checked']

In both cases using -1 as endpos is simpler.

Ezio requested further opinions, so here's mine. I don't think the current behavior makes sense; I doubt anyone actually expects a negative index to be squashed to 0, especially for endpos. I'm not certain that allowing negative indexes is really necessary, but seems nicer than raising an exception which would be the other acceptable option to me.

Note that changing the current behavior is a breaking change. For example someone can use pattern.findall(text, curpos-50, curpos+50) to search in the range ±50 characters from the current position. If negative positions change meaning, this will break a code for curpos < 50.

Navigation Menu

Repository navigation

re.finditer and re.findall should support negative end positions #52188

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

re.finditer and re.findall should support negative end positions #52188

Description

Linked PRs

Activity

tlynn commented on Feb 16, 2010

mrabarnett commented on Feb 17, 2010

BreamoreBoy commented on May 23, 2013

mrabarnett commented on May 23, 2013

mrabarnett commented on May 26, 2013

serhiy-storchaka commented on May 26, 2013

mrabarnett commented on May 26, 2013

rhettinger commented on Jul 13, 2019

ezio-melotti commented on Jul 14, 2019

ezio-melotti commented on Jul 14, 2019

serhiy-storchaka commented on Jul 14, 2019

anilbey commented on Jul 15, 2019

ezio-melotti commented on Jul 16, 2019

zware commented on Sep 9, 2019

serhiy-storchaka commented on Sep 9, 2019

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions