Skip to content

gh-84508: tool to generate cjk traditional chinese mappings #93272

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
corona10 merged 7 commits into from
Jun 11, 2022
Merged

gh-84508: tool to generate cjk traditional chinese mappings #93272

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
1fb94f7
gh-84508: tool to generate cjk traditional chinese mappings
sorcio May 26, 2022
0eae907
strip trailing whitespace
sorcio May 26, 2022
9a008bd
Apply suggestions from code review
sorcio May 28, 2022
bec0692
add unicode mapping files for big5 and cp950
sorcio May 28, 2022
39d77d9
generate mappings_tw.h and mappings_hk.h
sorcio May 28, 2022
5d0ec20
Revert "add unicode mapping files for big5 and cp950"
sorcio May 29, 2022
d831eac
Use https url for CP950.TXT
sorcio May 30, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Apply suggestions from code review 
Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
  • Loading branch information
@sorcio @corona10
sorcio and corona10 authored May 28, 2022
commit 9a008bdc9bc3cb6ae745d88af5372802501afb12
12 changes: 5 additions & 7 deletions Tools/unicode/genmap_tchinese.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,9 @@
#
# Original Author: Hye-Shik Chang <perky@FreeBSD.org>

@ezio-melotti ezio-melotti May 28, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW @hyeshik doesn't seem to be active on this repo, but he signed the CLA in 2005 (https://bugs.python.org/user1298), so it should be ok to include it here.

If this script is based on his script, you might want to clarify that in the comment.

@sorcio sorcio May 28, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ezio-melotti do you have a suggestion on how to amend the comment to clarify that? The mention of "Original Author" is what it currently looks like in the other genmap_* scripts, so I didn't take the liberty to sway from that style.

The original source (as for the other equivalent scripts) is https://github.com/BackupTheBerlios/cjkpython/blob/master/cjkcodecs/tools/genmap_tchinese.py which was originally under 2-clause BSD. I might have made a wrong assumption that it was ok to include in CPython, since the other scripts followed the same process?

@sorcio sorcio May 28, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(In case it matters, the script in this PR is heavily modified, so while it probably still counts as derivative, it would be easy to do a full rewrite)

#
from genmap_support import (
open_mapping_file,
loadmap,
DecodeMapWriter,
EncodeMapWriter,
BufferedFiller,
)
import os

from genmap_support import *


# ranges for (lead byte, follower byte)
Expand Down Expand Up @@ -173,6 +169,7 @@ def main_tw():
del cp950decmap[c1][c2]

with open('mappings_tw.h', 'w') as fp:
print_autogen(fp, os.path.basename(__file__))
write_big5_maps(fp, 'BIG5', 'big5', big5decmap, big5encmap)
Comment thread
sorcio marked this conversation as resolved.
write_big5_maps(fp, 'CP950', 'cp950ext', cp950decmap, cp950encmap)

Expand Down Expand Up @@ -217,6 +214,7 @@ def main_hkscs():
)
with open('mappings_hk.h', 'w') as fp:
print('Generating BIG5HKSCS decode map...')
print_autogen(fp, os.path.basename(__file__))
writer = DecodeMapWriter(fp, 'big5hkscs', hkscsdecmap)
Comment thread
sorcio marked this conversation as resolved.
writer.update_decode_map(BIG5HKSCS_C1, BIG5HKSCS_C2)
writer.generate()
Expand Down