re.sub[n] doesn't seem to handle /Z replacements correctly in all cases #54537

Activity

1. AlexanderSchmolck commented on Nov 5, 2010

   

   AlexanderSchmolckmannequin

   on Nov 5, 2010

   MannequinAuthor

   More actions

   In certain cases a zero-width /Z match that should be replaced isn't.

   An example might help:

   re.compile('(?m)(?P<trailing_ws>[ \t]+\r*$)|(?P<no_final_newline>(?<=[^\\n])\Z)').subn(lambda m:next('<'+k+'>' for k,v in m.groupdict().items() if v is not None), 'foobar ')

   this gives

   ('foobar<trailing_ws>', 1)

   I would have expected

   ('foobar<trailing_ws><no_final_newline>', 2)

   Contrast this with the following behavior:

   [m.span() for m in re.compile('(?P<trailing_ws>[ \t]+\r*$)|(?P<no_final_newline>(?<=[^\\n])\Z)', re.M).finditer('foobar ')]

   gives

   [(6, 7), (7, 7)]

   The matches are clearly not overlapping and the re module docs for sub say "Return the string obtained by replacing the leftmost non-overlapping occurrences of pattern in string by the replacement repl.", so I would have expected two replacements.

   This seems to be what perl is doing:

   echo -n 'foobar ' | perl -pe 's/(?m)(?P<trailing_ws>[ \t]+\r*$)|(?P<no_final_newline>(?<=[^\\n])\Z)/<$&>/g'

   gives
   foobar< ><>%
2. AlexanderSchmolck

   added

   topic-regex

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Nov 5, 2010
3. mrabarnett commented on Nov 5, 2010

   

   mrabarnettmannequin

   on Nov 5, 2010

   Mannequin

   More actions

   It's a bug caused by trying to avoid getting stuck when a zero-width match is found. Basically the fix is to advance one character after a zero-width match, but that doesn't always give the correct result.

   There are a number of related issues like issue bpo-1647489 ("zero-length match confuses re.finditer()").
4. BreamoreBoy commented on Sep 30, 2014

   

   BreamoreBoymannequin

   on Sep 30, 2014

   Mannequin

   More actions

   @serhiy can you take a look at this as I recall you've been doing some regex work?
5. animalize commented on Apr 27, 2019

   

   animalizemannequin

   on Apr 27, 2019

   Mannequin

   More actions

   This bug was fixed in Python 3.7, see bpo-32308.

   Python 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 23 2018, 23:31:17) [MSC v.1916 32 bit (Intel)] on win32
   >>> re.compile('(?m)(?P<trailing_ws>[ \t]+\r*$)|(?P<no_final_newline>(?<=[^\n])\Z)').subn(lambda m:next('<'+k+'>' for k,v in m.groupdict().items() if v is not None), 'foobar ')
   ('foobar<trailing_ws>', 1)
   
   Python 3.7.3rc1 (tags/v3.7.3rc1:69785b2127, Mar 12 2019, 22:37:55) [MSC v.1916 64 bit (AMD64)] on win32
   >>> re.compile('(?m)(?P<trailing_ws>[ \t]+\r*$)|(?P<no_final_newline>(?<=[^\n])\Z)').subn(lambda m:next('<'+k+'>' for k,v in m.groupdict().items() if v is not None), 'foobar ')
   ('foobar<trailing_ws><no_final_newline>', 2)
6. serhiy-storchaka

   closed this as completedon Apr 27, 2019
7. ezio-melotti

   transferred this issue fromon Apr 10, 2022