Skip to content

gh-74902: add unicode grapheme cluster break algorithm #2673

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Vermeille wants to merge 13 commits into from
Closed

gh-74902: add unicode grapheme cluster break algorithm #2673

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
b79f969
add unicodedata.grapheme_cluster_break()
Vermeille Jul 11, 2017
c9a4211
generate unicodedata.c.h with clinic
Vermeille Jul 13, 2017
7f56b78
patchcheck
Vermeille Jul 13, 2017
a47de54
add the grapheme cluster break automaton
Vermeille Jul 13, 2017
c152171
add my name to Misc/ACKS
Vermeille Jul 14, 2017
b103be7
code review fixes
Vermeille Aug 3, 2017
c9848e2
rename break_graphemes to iter_graphemes
Vermeille Aug 3, 2017
2dee91e
make GraphemeClusterIterator a GC type
Vermeille Aug 3, 2017
a5b3c10
allow iterating only over a range of indices
Vermeille Jan 10, 2018
ba1e9b7
Merge branch 'main' into grapheme_cluster_break
serhiy-storchaka Dec 17, 2025
ae06154
Add some tests.
serhiy-storchaka Dec 17, 2025
1965ed6
Save Grapheme_Cluster_Break for unassigned code points.
serhiy-storchaka Dec 17, 2025
164d19b
Make "Any" the first entry.
serhiy-storchaka Dec 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
add unicodedata.grapheme_cluster_break()
  • Loading branch information
@Vermeille
Vermeille committed Jul 11, 2017
commit b79f9699fadf6a621311ca5feb4a7000938a8cbe
28 changes: 28 additions & 0 deletions Modules/clinic/unicodedata.c.h
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

22 changes: 22 additions & 0 deletions Modules/unicodedata.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ typedef struct {
const unsigned char east_asian_width; /* index into
_PyUnicode_EastAsianWidth */
const unsigned char normalization_quick_check; /* see is_normalized() */
const unsigned char grapheme_cluster_break;
} _PyUnicode_DatabaseRecord;

typedef struct change_record {
Expand Down Expand Up @@ -382,6 +383,26 @@ unicodedata_UCD_east_asian_width_impl(PyObject *self, int chr)
return PyUnicode_FromString(_PyUnicode_EastAsianWidthNames[index]);
}

/*[clinic input]
unicodedata.UCD.grapheme_cluster_break

self: self
chr: int(accept={str})
/

Returns the east asian width assigned to the character chr as string.
[clinic start generated code]*/

static PyObject *
unicodedata_UCD_grapheme_cluster_break_impl(PyObject *self, int chr)
/*[clinic end generated code: output=484e8537d9ee8197 input=c4854798aab026e0]*/
{
int index;
Py_UCS4 c = (Py_UCS4)chr;
index = (int) _getrecord_ex(c)->grapheme_cluster_break;
return PyUnicode_FromString(_PyUnicode_GraphemeBreakProperty[index]);
}

/*[clinic input]
unicodedata.UCD.decomposition

Expand Down Expand Up @@ -1264,6 +1285,7 @@ static PyMethodDef unicodedata_functions[] = {
UNICODEDATA_UCD_COMBINING_METHODDEF
UNICODEDATA_UCD_MIRRORED_METHODDEF
UNICODEDATA_UCD_EAST_ASIAN_WIDTH_METHODDEF
UNICODEDATA_UCD_GRAPHEME_CLUSTER_BREAK_METHODDEF
UNICODEDATA_UCD_DECOMPOSITION_METHODDEF
UNICODEDATA_UCD_NAME_METHODDEF
UNICODEDATA_UCD_LOOKUP_METHODDEF
Expand Down
Loading