Incorrect charset range handling with ignore case flag? #47761

Activity

1. mrabarnett commented on Aug 6, 2008

   

   mrabarnettmannequin

   on Aug 6, 2008

   MannequinAuthor

   More actions

   While working on the regex code in sre_compile.py I came across the
   following code in the handling of charset ranges in _optimize_charset:

       for i in range(fixup(av[0]), fixup(av[1])+1):
           charmap[i] = 1

   The function fixup converts the ends of the range to lower case if the
   ignore-case flag is present. The problem with this approach is
   illustrated below:

   >>> import re
   >>> print re.match(r'[9-A]', 'A')
   <_sre.SRE_Match object at 0x00A78058>
   >>> print re.match(r'[9-A]', 'a')
   None
   >>> print re.match(r'[9-A]', '_')
   None
   >>> print re.match(r'[9-A]', 'A', re.IGNORECASE)
   <_sre.SRE_Match object at 0x00D0BFA8>
   >>> print re.match(r'[9-A]', 'a', re.IGNORECASE)
   <_sre.SRE_Match object at 0x00A78058>
   >>> print re.match(r'[9-A]', '_', re.IGNORECASE)
   <_sre.SRE_Match object at 0x00D0BFA8>
   >>> 

   '_' doesn't lie between '9' and 'A', but it does lie between '9' and 'a'.

   Surely the ignore-case flag should not affect whether non-letters are
   matched or not?
2. mrabarnett

   added

   topic-regex

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Aug 6, 2008
3. timehorse commented on Sep 24, 2008

   

   timehorsemannequin

   on Sep 24, 2008

   Mannequin

   More actions

   I think this is even more complicated when you consider that
   localization my be an issue. Consider "Á": is this grammatically before
   "A" or after "a"? From a character set point of view, it is typically
   after "a" but when Locale is taken into account, all that is done is
   there is a change to relative ordering, so Á appears somewhere before A
   and B. But when this is done, does that mean that [9-Á] is going to
   cover ALL uppercase and ALL lowercase and ALL characters with ord from
   91 to 96 and 123 to 127 and all kinds of other UNICODE symbols? And how
   will this effect case-insensitivity.

   In a sense, I think it may only be safe to say that character class
   ranges are ONLY appropriate over Alphabetic character ranges or numeric
   character ranges, since the order of the ASCII symbols between 0 and 47,
   56 and 64, 91 adn 96 and 123 and 127, though well-defined, are none the
   less implementation dependent. When we bring UNICODE into this, things
   get even more befuddled with some Latin characters in Latin-1, some in
   Latin-2, Cyrillic, Hebrew, Arabic, Chinese, Japanese and Korean
   character sets just to name a few of the most common! And how does a
   total ordering of characters apply to them?

   In the end, I think it's just dangerous to define character group ranges
   that span the gap BETWEEN numbers and alphabetics. Instead, I think a
   better solution is simply to implement Emacs / Perl style named
   character classes as in bpo-2636 sub-item 8.

   I do agree this is a problem, but as I see it, the solution may not be
   that simple, especially in a UNICODE world.
4. ezio-melotti commented on Feb 20, 2009

   

   ezio-melotti

   on Feb 20, 2009

   Member

   More actions

   I'd close this as "won't fix", because (IMHO) ranges like [9-A]
   shouldn't be used at all, so I won't expect it to work properly.

   FWIW Perl doesn't seem to match the '_', even with the 'i' flag. Tested
   with: perl -e '$s = ("_" =~ /[9-A]/); print $s' and perl -e '$s = ("_"
   =~ /[9-A]/i); print $s'. It matches ":" with [9-A] and "_" with [9-a]
   though (both without the 'i' flag).
5. mrabarnett commented on Feb 20, 2009

   

   mrabarnettmannequin

   on Feb 20, 2009

   MannequinAuthor

   More actions

   "[9-A]" is equivalent to "[9:;<=>?@A]", or should be.

   It'll be fixed in issue bpo-2636.
6. ezio-melotti commented on Feb 20, 2009

   

   ezio-melotti

   on Feb 20, 2009

   Member

   More actions

   If there's already a patch, then it's fine (and useful for ranges of
   Unicode chars), but I wouldn't like to find a range like [9-A] around ;)
7. terryjreedy commented on Aug 3, 2010

   

   terryjreedy

   on Aug 3, 2010

   Member

   More actions

   EM and MB seemed to agree on closing this.
8. terryjreedy

   closed this as completedon Aug 3, 2010
9. serhiy-storchaka commented on Nov 8, 2014

   

   serhiy-storchaka

   on Nov 8, 2014

   Member

   More actions

   Fixed in bpo-17381 (which has more realistic example than [9-A]).
10. ezio-melotti

    transferred this issue fromon Apr 10, 2022