Repository navigation
Request for property support in Python re lib #56943
Description
| BPO | 12734 |
|---|---|
| Nosy | @gvanrossum, @ezio-melotti, @bitdancer, @jayvdb |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2011-08-11.20:14:13.511>
labels = ['expert-regex', 'type-feature']
title = 'Request for property support in Python re lib'
updated_at = <Date 2021-01-26.09:56:36.211>
user = 'https://bugs.python.org/tchrist'bugs.python.org fields:
activity = <Date 2021-01-26.09:56:36.211>
actor = 'jayvdb'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Regular Expressions']
creation = <Date 2011-08-11.20:14:13.511>
creator = 'tchrist'
dependencies = []
files = []
hgrepos = []
issue_num = 12734
keywords = []
message_count = 7.0
messages = ['141925', '141931', '141934', '142113', '142141', '142142', '143042']
nosy_count = 8.0
nosy_names = ['gvanrossum', 'ezio.melotti', 'mrabarnett', 'Arfrever', 'r.david.murray', 'tchrist', 'jayvdb', 'Socob']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue12734'
versions = ['Python 3.4']
Activity
tchrist commentedon Aug 11, 2011
Python supports no Unicode properties in its re library, making it unsuitable for work with Unicode. This is therefore a formal request for the Python re library to support Unicode properties.
The eleven properties required by Unicode Technical Report #18's RL1.2 are the bare minimum which must be added to make it possible to use Python reguyar expressions on Unicode.
The proposed RL2.7 on Full Properties is even better. That is found at
http://unicode.org/reports/tr18/proposed.html#Full_Properties
Although by the time you read this, it will have been made an official part of tr18.
Matthew Barnett's replacement library for re, called regex, support 67 Unicode properties at last count, including the strongly recommended loose matching.
The standard re library needs to be spiffed up to make it suitable for Unicode processing; it is not currently usable for that due to this missing functionality. I quote from the Level 1 conformance requirement of tr18:
pass RL1.1 Hex Notation
fail RL1.2 Properties
fail RL1.2a Compatibility Properties
fail RL1.3 Subtraction and Intersection
fail RL1.4 Simple Word Boundaries
fail RL1.5 Simple Loose Matches
fail RL1.6 Line Boundaries
fail RL1.7 Supplementary Code Points
(withdrawn) RL2.1 Canonical Equivalents
fail RL2.2 Extended Grapheme Clusters
fail RL2.3 Default Word Boundaries
fail RL2.4 Default Case Conversion
pass RL2.5 Name Properties
fail RL2.6 Wildcards in Property Values
fail RL2.7 Full Properties
I won’t even talk about Level 3.
ICU, Perl, and Java7 all meet Level One conformance requirements with several Level 2 requirements also met. It is important for Python to meet the Unicode Standard in this so that people can use Python for regex matching Unicode text. They currently cannot usefully do so per the requirements of tr18.
bitdancer commentedon Aug 12, 2011
I think the only way re is going to get "spiffed up" is by replacing it with Matthew's library. This is a goal, but I'm not sure where exactly we are in the process. The more Matthew's code gets tested (especially for compatibility with the current re API), the closer we will be to that goal.
tchrist commentedon Aug 12, 2011
I've been a lot of testing of Matthew's regex library against UTS#18 issues, but only somewhat incidentally testing re. To use regex, one has to accept that certain things will work differently than they work in re, because he is following Unicode definitions for things like casefolding.
But I doubt that is the sort of difference you are talking about. One of the things that Java, Go, and Perl all do is run regression tests against the whole Unicode Character Database to make sure nothing gets hosed, missed, or otherwise out of sync. That might a sort of regression test you might like to add.
ezio-melotti commentedon Aug 15, 2011
This indeed should be "fixed" by replacing 're' with 'regex'. So I would suggest to focus your tests on 'regex' and report them there so that possible bugs gets fixed and tested before we include the module in the stdlib.
tchrist commentedon Aug 15, 2011
Sorry I didn't include a test case. Hope this makes up for it. If not, please tell me how to write better test cases. :(
Yeah ok, so I'm a bit persnickety or even unorthodox about my vertical alignment, but it really helps to make what is different from one to line to the next stand out if the parts that are the same from line to line are at the same column every time.
tchrist commentedon Aug 15, 2011
Oh whoops, that was the long ticket. Shall I reupload to the right number?
gvanrossum commentedon Aug 26, 2011
+1 on adding the feature to 3.3 in whichever way makes sense.