Skip to content

bpo-34294: re module, fix wrong capturing groups in rare cases #11546

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
serhiy-storchaka merged 5 commits into from Feb 18, 2019
Merged

bpo-34294: re module, fix wrong capturing groups in rare cases #11546

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b456851
re module, reset capturing groups inside SRE(search) function.
wjssz Jan 14, 2019
314cb49
put `RESET_CAPTURE_GROUP();` after `ptr++`
wjssz Jan 26, 2019
a665528
remove unnecessary change
wjssz Feb 9, 2019
1afd7ba
reword news entry
wjssz Feb 18, 2019
fcf830c
format
wjssz Feb 18, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Doc/library/re.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -368,6 +368,8 @@ The special characters are:
``(?#...)``
A comment; the contents of the parentheses are simply ignored.

.. index:: single: (?=; in regular expressions

``(?=...)``
Matches if ``...`` matches next, but doesn't consume any of the string. This is
called a :dfn:`lookahead assertion`. For example, ``Isaac (?=Asimov)`` will match
Expand Down
34 changes: 34 additions & 0 deletions Lib/test/test_re.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2067,6 +2067,40 @@ def test_bug_29444(self):
self.assertEqual(m.group(), b'xyz')
self.assertEqual(m2.group(), b'')

def test_bug_34294(self):
# Issue 34294: wrong capturing groups

# exists since Python 2
s = "a\tx"
p = r"\b(?=(\t)|(x))x"
self.assertEqual(re.search(p, s).groups(), (None, 'x'))

# introduced in Python 3.7.0
s = "ab"
p = r"(?=(.)(.)?)"
self.assertEqual(re.findall(p, s),
[('a', 'b'), ('b', '')])
self.assertEqual([m.groups() for m in re.finditer(p, s)],
[('a', 'b'), ('b', None)])

# test-cases provided by issue34294, introduced in Python 3.7.0
p = r"(?=<(?P<tag>\w+)/?>(?:(?P<text>.+?)</(?P=tag)>)?)"
s = "<test><foo2/></test>"
self.assertEqual(re.findall(p, s),
[('test', '<foo2/>'), ('foo2', '')])
self.assertEqual([m.groupdict() for m in re.finditer(p, s)],
[{'tag': 'test', 'text': '<foo2/>'},
{'tag': 'foo2', 'text': None}])
s = "<test>Hello</test><foo/>"
self.assertEqual([m.groupdict() for m in re.finditer(p, s)],
[{'tag': 'test', 'text': 'Hello'},
{'tag': 'foo', 'text': None}])
s = "<test>Hello</test><foo/><foo/>"
self.assertEqual([m.groupdict() for m in re.finditer(p, s)],
[{'tag': 'test', 'text': 'Hello'},
{'tag': 'foo', 'text': None},
{'tag': 'foo', 'text': None}])


class PatternReprTests(unittest.TestCase):
def check(self, pattern, expected):
Expand Down
1 change: 1 addition & 0 deletions Misc/NEWS.d/next/Library/2019-01-14-11-53-10.bpo-34294.3JFdg2.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
re module, fix wrong capturing groups in rare cases. :func:`re.search`, :func:`re.findall`, :func:`re.sub` and other functions that scan through string looking for a match, should reset capturing groups between two match attempts.
Comment thread
This conversation was marked as resolved.
Outdated
2 changes: 1 addition & 1 deletion Modules/_sre.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -340,7 +340,7 @@ _sre_unicode_tolower_impl(PyObject *module, int character)
LOCAL(void)
state_reset(SRE_STATE* state)
{
/* FIXME: dynamic! */
/* state->mark will be set to 0 in SRE_OP_MARK dynamically. */
/*memset(state->mark, 0, sizeof(*state->mark) * SRE_MARK_SIZE);*/

state->lastmark = -1;
Expand Down
8 changes: 8 additions & 0 deletions Modules/sre_lib.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1363,6 +1363,10 @@ SRE(match)(SRE_STATE* state, SRE_CODE* pattern, int toplevel)
return ret; /* should never get here */
}

/* need to reset capturing groups between two SRE(match) callings in loops */
#define RESET_CAPTURE_GROUP() \
do { state->lastmark = state->lastindex = -1; } while (0)

LOCAL(Py_ssize_t)
SRE(search)(SRE_STATE* state, SRE_CODE* pattern)
{
Expand Down Expand Up @@ -1440,6 +1444,7 @@ SRE(search)(SRE_STATE* state, SRE_CODE* pattern)
if (status != 0)
return status;
++ptr;
RESET_CAPTURE_GROUP();
}
return 0;
}
Expand Down Expand Up @@ -1487,6 +1492,7 @@ SRE(search)(SRE_STATE* state, SRE_CODE* pattern)
/* close but no cigar -- try again */
if (++ptr >= end)
return 0;
RESET_CAPTURE_GROUP();
}
i = overlap[i];
} while (i != 0);
Expand All @@ -1510,6 +1516,7 @@ SRE(search)(SRE_STATE* state, SRE_CODE* pattern)
if (status != 0)
break;
ptr++;
RESET_CAPTURE_GROUP();
}
} else {
/* general case */
Expand All @@ -1520,6 +1527,7 @@ SRE(search)(SRE_STATE* state, SRE_CODE* pattern)
state->must_advance = 0;
while (status == 0 && ptr < end) {
ptr++;
RESET_CAPTURE_GROUP();
TRACE(("|%p|%p|SEARCH\n", pattern, ptr));
state->start = state->ptr = ptr;
status = SRE(match)(state, pattern, 0);
Expand Down