Mercurial > cpython-lru_cache-weakref

annotate Tools/unicode/makeunicodedata.py @ 84007:dbfbdf2b5c19 3.3

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Fix #17967: For ftp urls CWD to target instead of hopping to each directory towards target. This fixes a bug where target is accessible, but parent directories are restricted.
author Senthil Kumaran <senthil@uthcode.com>
date Sat, 01 Jun 2013 08:27:06 -0700
parents 5b71f5891c54
children c57d99a8fc42 dfe991036531
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1 #
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
2 # (re)generate unicode property and type databases
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
3 #
26397
b27c1cad77ea Add unidata_version. Bump generator version number.
Martin v. Löwis <martin@v.loewis.de>
parents: 26394
diff changeset
4 # this script converts a unicode 3.2 database file to
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
5 # Modules/unicodedata_db.h, Modules/unicodename_db.h,
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
6 # and Objects/unicodetype_db.h
14835
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
7 #
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
8 # history:
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
9 # 2000-09-24 fl created (based on bits and pieces from unidb)
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
10 # 2000-09-25 fl merged tim's splitbin fixes, separate decomposition table
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
11 # 2000-09-25 fl added character type table
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
12 # 2000-09-26 fl added LINEBREAK, DECIMAL, and DIGIT flags/fields (2.0)
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
13 # 2000-11-03 fl expand first/last ranges
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
14 # 2001-01-19 fl added character name tables (2.1)
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
15 # 2001-01-21 fl added decomp compression; dynamic phrasebook threshold
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
16 # 2002-09-11 wd use string methods
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
17 # 2002-10-18 mvl update to Unicode 3.2
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
18 # 2002-10-22 mvl generate NFC tables
26394
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
19 # 2002-11-24 mvl expand all ranges, sort names version-independently
26397
b27c1cad77ea Add unidata_version. Bump generator version number.
Martin v. Löwis <martin@v.loewis.de>
parents: 26394
diff changeset
20 # 2002-11-25 mvl add UNIDATA_VERSION
32197
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 26397
diff changeset
21 # 2004-05-29 perky add east asian width information
36940
e4465d2737c4 Add changelog entry.
Martin v. Löwis <martin@v.loewis.de>
parents: 36937
diff changeset
22 # 2006-03-10 mvl update to Unicode 4.1; add UCD 3.2 delta
48427
4e6859c6d38c Issue #3282: str.isprintable() should return False for undefined Unicode characters.
Georg Brandl <georg@python.org>
parents: 47680
diff changeset
23 # 2008-06-11 gb add PRINTABLE_MASK for Atsuo Ishimoto's ascii() patch
73044
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
24 # 2011-10-21 ezio add support for name aliases and named sequences
74338
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
25 # 2012-01 benjamin add full case mappings
14835
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
26 #
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
27 # written by Fredrik Lundh (fredrik@pythonware.com)
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
28 #
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
29
73044
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
30 import os
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
31 import sys
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
32 import zipfile
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
33
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
34 from textwrap import dedent
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
35
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
36 SCRIPT = sys.argv[0]
64146
8f97c64b2333 #5127: Even on narrow unicode builds, the C functions that access the Unicode
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 60086
diff changeset
37 VERSION = "3.2"
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
38
26397
b27c1cad77ea Add unidata_version. Bump generator version number.
Martin v. Löwis <martin@v.loewis.de>
parents: 26394
diff changeset
39 # The Unicode Database
75107
cd21e6d30a1a update to Unicode 6.1
Benjamin Peterson <benjamin@python.org>
parents: 74423
diff changeset
40 UNIDATA_VERSION = "6.1.0"
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
41 UNICODE_DATA = "UnicodeData%s.txt"
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
42 COMPOSITION_EXCLUSIONS = "CompositionExclusions%s.txt"
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
43 EASTASIAN_WIDTH = "EastAsianWidth%s.txt"
65217
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
44 UNIHAN = "Unihan%s.zip"
42595
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 42427
diff changeset
45 DERIVED_CORE_PROPERTIES = "DerivedCoreProperties%s.txt"
53554
423f8f9b07b1 Merged revisions 72054 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 53402
diff changeset
46 DERIVEDNORMALIZATION_PROPS = "DerivedNormalizationProps%s.txt"
60086
bf3c48f2417c Merged revisions 79494,79496 via svnmerge from
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59720
diff changeset
47 LINE_BREAK = "LineBreak%s.txt"
73044
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
48 NAME_ALIASES = "NameAliases%s.txt"
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
49 NAMED_SEQUENCES = "NamedSequences%s.txt"
74338
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
50 SPECIAL_CASING = "SpecialCasing%s.txt"
74405
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
51 CASE_FOLDING = "CaseFolding%s.txt"
73044
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
52
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
53 # Private Use Areas -- in planes 1, 15, 16
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
54 PUA_1 = range(0xE000, 0xF900)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
55 PUA_15 = range(0xF0000, 0xFFFFE)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
56 PUA_16 = range(0x100000, 0x10FFFE)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
57
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
58 # we use this ranges of PUA_15 to store name aliases and named sequences
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
59 NAME_ALIASES_START = 0xF0000
75107
cd21e6d30a1a update to Unicode 6.1
Benjamin Peterson <benjamin@python.org>
parents: 74423
diff changeset
60 NAMED_SEQUENCES_START = 0xF0200
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
61
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
62 old_versions = ["3.2.0"]
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
63
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
64 CATEGORY_NAMES = [ "Cn", "Lu", "Ll", "Lt", "Mn", "Mc", "Me", "Nd",
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
65 "Nl", "No", "Zs", "Zl", "Zp", "Cc", "Cf", "Cs", "Co", "Cn", "Lm",
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
66 "Lo", "Pc", "Pd", "Ps", "Pe", "Pi", "Pf", "Po", "Sm", "Sc", "Sk",
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
67 "So" ]
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
68
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
69 BIDIRECTIONAL_NAMES = [ "", "L", "LRE", "LRO", "R", "AL", "RLE", "RLO",
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
70 "PDF", "EN", "ES", "ET", "AN", "CS", "NSM", "BN", "B", "S", "WS",
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
71 "ON" ]
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
72
32951
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32197
diff changeset
73 EASTASIANWIDTH_NAMES = [ "F", "H", "W", "Na", "A", "N" ]
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32197
diff changeset
74
60086
bf3c48f2417c Merged revisions 79494,79496 via svnmerge from
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59720
diff changeset
75 MANDATORY_LINE_BREAKS = [ "BK", "CR", "LF", "NL" ]
bf3c48f2417c Merged revisions 79494,79496 via svnmerge from
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59720
diff changeset
76
14852
463faf01685e unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14849
diff changeset
77 # note: should match definitions in Objects/unicodectype.c
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
78 ALPHA_MASK = 0x01
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
79 DECIMAL_MASK = 0x02
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
80 DIGIT_MASK = 0x04
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
81 LOWER_MASK = 0x08
14852
463faf01685e unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14849
diff changeset
82 LINEBREAK_MASK = 0x10
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
83 SPACE_MASK = 0x20
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
84 TITLE_MASK = 0x40
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
85 UPPER_MASK = 0x80
42595
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 42427
diff changeset
86 XID_START_MASK = 0x100
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 42427
diff changeset
87 XID_CONTINUE_MASK = 0x200
48427
4e6859c6d38c Issue #3282: str.isprintable() should return False for undefined Unicode characters.
Georg Brandl <georg@python.org>
parents: 47680
diff changeset
88 PRINTABLE_MASK = 0x400
74338
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
89 NUMERIC_MASK = 0x800
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
90 CASE_IGNORABLE_MASK = 0x1000
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
91 CASED_MASK = 0x2000
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
92 EXTENDED_CASE_MASK = 0x4000
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
93
66392
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 65217
diff changeset
94 # these ranges need to match unicodedata.c:is_unified_ideograph
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 65217
diff changeset
95 cjk_ranges = [
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 65217
diff changeset
96 ('3400', '4DB5'),
75107
cd21e6d30a1a update to Unicode 6.1
Benjamin Peterson <benjamin@python.org>
parents: 74423
diff changeset
97 ('4E00', '9FCC'),
66392
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 65217
diff changeset
98 ('20000', '2A6D6'),
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 65217
diff changeset
99 ('2A700', '2B734'),
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 65217
diff changeset
100 ('2B740', '2B81D')
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 65217
diff changeset
101 ]
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
102
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
103 def maketables(trace=0):
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
104
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
105 print("--- Reading", UNICODE_DATA % "", "...")
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
106
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
107 version = ""
65217
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
108 unicode = UnicodeData(UNIDATA_VERSION)
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
109
47611
da47e7e135ae #2630: Implement PEP 3138.
Georg Brandl <georg@python.org>
parents: 47463
diff changeset
110 print(len(list(filter(None, unicode.table))), "characters")
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
111
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
112 for version in old_versions:
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
113 print("--- Reading", UNICODE_DATA % ("-"+version), "...")
66392
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 65217
diff changeset
114 old_unicode = UnicodeData(version, cjk_check=False)
47611
da47e7e135ae #2630: Implement PEP 3138.
Georg Brandl <georg@python.org>
parents: 47463
diff changeset
115 print(len(list(filter(None, old_unicode.table))), "characters")
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
116 merge_old_version(version, unicode, old_unicode)
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
117
16091
45c065668ea5 Unicode nits: Don't include unicodedatabase.h no more. And make sure
Fredrik Lundh <fredrik@pythonware.com>
parents: 16090
diff changeset
118 makeunicodename(unicode, trace)
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
119 makeunicodedata(unicode, trace)
16091
45c065668ea5 Unicode nits: Don't include unicodedatabase.h no more. And make sure
Fredrik Lundh <fredrik@pythonware.com>
parents: 16090
diff changeset
120 makeunicodetype(unicode, trace)
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
121
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
122 # --------------------------------------------------------------------
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
123 # unicode character properties
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
124
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
125 def makeunicodedata(unicode, trace):
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
126
53554
423f8f9b07b1 Merged revisions 72054 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 53402
diff changeset
127 dummy = (0, 0, 0, 0, 0, 0)
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
128 table = [dummy]
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
129 cache = {0: dummy}
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
130 index = [0] * len(unicode.chars)
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
131
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
132 FILE = "Modules/unicodedata_db.h"
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
133
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
134 print("--- Preparing", FILE, "...")
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
135
14835
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
136 # 1) database properties
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
137
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
138 for char in unicode.chars:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
139 record = unicode.table[char]
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
140 if record:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
141 # extract database properties
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
142 category = CATEGORY_NAMES.index(record[2])
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
143 combining = int(record[3])
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
144 bidirectional = BIDIRECTIONAL_NAMES.index(record[4])
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
145 mirrored = record[9] == "Y"
32951
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32197
diff changeset
146 eastasianwidth = EASTASIANWIDTH_NAMES.index(record[15])
53554
423f8f9b07b1 Merged revisions 72054 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 53402
diff changeset
147 normalizationquickcheck = record[17]
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
148 item = (
53554
423f8f9b07b1 Merged revisions 72054 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 53402
diff changeset
149 category, combining, bidirectional, mirrored, eastasianwidth,
423f8f9b07b1 Merged revisions 72054 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 53402
diff changeset
150 normalizationquickcheck
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
151 )
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
152 # add entry to index and item tables
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
153 i = cache.get(item)
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
154 if i is None:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
155 cache[item] = i = len(table)
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
156 table.append(item)
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
157 index[char] = i
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
158
14835
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
159 # 2) decomposition data
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
160
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
161 decomp_data = [0]
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
162 decomp_prefix = [""]
14835
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
163 decomp_index = [0] * len(unicode.chars)
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
164 decomp_size = 0
14835
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
165
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
166 comp_pairs = []
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
167 comp_first = [None] * len(unicode.chars)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
168 comp_last = [None] * len(unicode.chars)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
169
14835
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
170 for char in unicode.chars:
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
171 record = unicode.table[char]
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
172 if record:
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
173 if record[5]:
25219
8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113
Walter Dörwald <walter@livinglogic.de>
parents: 16091
diff changeset
174 decomp = record[5].split()
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
175 if len(decomp) > 19:
42715
e4ade16a694f Convert raise statements in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42595
diff changeset
176 raise Exception("character %x has a decomposition too large for nfd_nfkd" % char)
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
177 # prefix
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
178 if decomp[0][0] == "<":
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
179 prefix = decomp.pop(0)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
180 else:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
181 prefix = ""
14835
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
182 try:
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
183 i = decomp_prefix.index(prefix)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
184 except ValueError:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
185 i = len(decomp_prefix)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
186 decomp_prefix.append(prefix)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
187 prefix = i
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
188 assert prefix < 256
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
189 # content
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 45049
diff changeset
190 decomp = [prefix + (len(decomp)<<8)] + [int(s, 16) for s in decomp]
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
191 # Collect NFC pairs
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
192 if not prefix and len(decomp) == 3 and \
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
193 char not in unicode.exclusions and \
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
194 unicode.table[decomp[1]][3] == "0":
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
195 p, l, r = decomp
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
196 comp_first[l] = 1
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
197 comp_last[r] = 1
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
198 comp_pairs.append((l,r,char))
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
199 try:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
200 i = decomp_data.index(decomp)
14835
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
201 except ValueError:
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
202 i = len(decomp_data)
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
203 decomp_data.extend(decomp)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
204 decomp_size = decomp_size + len(decomp) * 2
14835
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
205 else:
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
206 i = 0
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
207 decomp_index[char] = i
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
208
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
209 f = l = 0
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
210 comp_first_ranges = []
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
211 comp_last_ranges = []
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
212 prev_f = prev_l = None
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
213 for i in unicode.chars:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
214 if comp_first[i] is not None:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
215 comp_first[i] = f
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
216 f += 1
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
217 if prev_f is None:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
218 prev_f = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
219 elif prev_f[1]+1 == i:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
220 prev_f = prev_f[0],i
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
221 else:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
222 comp_first_ranges.append(prev_f)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
223 prev_f = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
224 if comp_last[i] is not None:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
225 comp_last[i] = l
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
226 l += 1
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
227 if prev_l is None:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
228 prev_l = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
229 elif prev_l[1]+1 == i:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
230 prev_l = prev_l[0],i
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
231 else:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
232 comp_last_ranges.append(prev_l)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
233 prev_l = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
234 comp_first_ranges.append(prev_f)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
235 comp_last_ranges.append(prev_l)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
236 total_first = f
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
237 total_last = l
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
238
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
239 comp_data = [0]*(total_first*total_last)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
240 for f,l,char in comp_pairs:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
241 f = comp_first[f]
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
242 l = comp_last[l]
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
243 comp_data[f*total_last+l] = char
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
244
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
245 print(len(table), "unique properties")
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
246 print(len(decomp_prefix), "unique decomposition prefixes")
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
247 print(len(decomp_data), "unique decomposition entries:", end=' ')
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
248 print(decomp_size, "bytes")
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
249 print(total_first, "first characters in NFC")
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
250 print(total_last, "last characters in NFC")
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
251 print(len(comp_pairs), "NFC pairs")
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
252
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
253 print("--- Writing", FILE, "...")
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
254
15339
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
255 fp = open(FILE, "w")
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
256 print("/* this file was generated by %s %s */" % (SCRIPT, VERSION), file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
257 print(file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
258 print('#define UNIDATA_VERSION "%s"' % UNIDATA_VERSION, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
259 print("/* a list of unique database records */", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
260 print("const _PyUnicode_DatabaseRecord _PyUnicode_Database_Records[] = {", file=fp)
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
261 for item in table:
53554
423f8f9b07b1 Merged revisions 72054 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 53402
diff changeset
262 print(" {%d, %d, %d, %d, %d, %d}," % item, file=fp)
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
263 print("};", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
264 print(file=fp)
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
265
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
266 print("/* Reindexing of NFC first characters. */", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
267 print("#define TOTAL_FIRST",total_first, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
268 print("#define TOTAL_LAST",total_last, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
269 print("struct reindex{int start;short count,index;};", file=fp)
47680
b5e02775fffc Merged revisions 64226 via svnmerge from
Martin v. Löwis <martin@v.loewis.de>
parents: 47611
diff changeset
270 print("static struct reindex nfc_first[] = {", file=fp)
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
271 for start,end in comp_first_ranges:
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
272 print(" { %d, %d, %d}," % (start,end-start,comp_first[start]), file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
273 print(" {0,0,0}", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
274 print("};\n", file=fp)
47680
b5e02775fffc Merged revisions 64226 via svnmerge from
Martin v. Löwis <martin@v.loewis.de>
parents: 47611
diff changeset
275 print("static struct reindex nfc_last[] = {", file=fp)
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
276 for start,end in comp_last_ranges:
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
277 print(" { %d, %d, %d}," % (start,end-start,comp_last[start]), file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
278 print(" {0,0,0}", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
279 print("};\n", file=fp)
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
280
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
281 # FIXME: <fl> the following tables could be made static, and
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
282 # the support code moved into unicodedatabase.c
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
283
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
284 print("/* string literals */", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
285 print("const char *_PyUnicode_CategoryNames[] = {", file=fp)
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
286 for name in CATEGORY_NAMES:
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
287 print(" \"%s\"," % name, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
288 print(" NULL", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
289 print("};", file=fp)
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
290
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
291 print("const char *_PyUnicode_BidirectionalNames[] = {", file=fp)
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
292 for name in BIDIRECTIONAL_NAMES:
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
293 print(" \"%s\"," % name, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
294 print(" NULL", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
295 print("};", file=fp)
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
296
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
297 print("const char *_PyUnicode_EastAsianWidthNames[] = {", file=fp)
32951
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32197
diff changeset
298 for name in EASTASIANWIDTH_NAMES:
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
299 print(" \"%s\"," % name, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
300 print(" NULL", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
301 print("};", file=fp)
32951
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32197
diff changeset
302
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
303 print("static const char *decomp_prefix[] = {", file=fp)
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
304 for name in decomp_prefix:
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
305 print(" \"%s\"," % name, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
306 print(" NULL", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
307 print("};", file=fp)
14835
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
308
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
309 # split record index table
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
310 index1, index2, shift = splitbins(index, trace)
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
311
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
312 print("/* index tables for the database records */", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
313 print("#define SHIFT", shift, file=fp)
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
314 Array("index1", index1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
315 Array("index2", index2).dump(fp, trace)
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
316
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
317 # split decomposition index table
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
318 index1, index2, shift = splitbins(decomp_index, trace)
14835
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
319
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
320 print("/* decomposition data */", file=fp)
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
321 Array("decomp_data", decomp_data).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
322
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
323 print("/* index tables for the decomposition data */", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
324 print("#define DECOMP_SHIFT", shift, file=fp)
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
325 Array("decomp_index1", index1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
326 Array("decomp_index2", index2).dump(fp, trace)
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
327
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
328 index, index2, shift = splitbins(comp_data, trace)
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
329 print("/* NFC pairs */", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
330 print("#define COMP_SHIFT", shift, file=fp)
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
331 Array("comp_index", index).dump(fp, trace)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
332 Array("comp_data", index2).dump(fp, trace)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
333
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
334 # Generate delta tables for old versions
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
335 for version, table, normalization in unicode.changed:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
336 cversion = version.replace(".","_")
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
337 records = [table[0]]
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
338 cache = {table[0]:0}
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
339 index = [0] * len(table)
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
340 for i, record in enumerate(table):
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
341 try:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
342 index[i] = cache[record]
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
343 except KeyError:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
344 index[i] = cache[record] = len(records)
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
345 records.append(record)
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
346 index1, index2, shift = splitbins(index, trace)
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
347 print("static const change_record change_records_%s[] = {" % cversion, file=fp)
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
348 for record in records:
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
349 print("\t{ %s }," % ", ".join(map(str,record)), file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
350 print("};", file=fp)
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
351 Array("changes_%s_index" % cversion, index1).dump(fp, trace)
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
352 Array("changes_%s_data" % cversion, index2).dump(fp, trace)
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
353 print("static const change_record* get_change_%s(Py_UCS4 n)" % cversion, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
354 print("{", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
355 print("\tint index;", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
356 print("\tif (n >= 0x110000) index = 0;", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
357 print("\telse {", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
358 print("\t\tindex = changes_%s_index[n>>%d];" % (cversion, shift), file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
359 print("\t\tindex = changes_%s_data[(index<<%d)+(n & %d)];" % \
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
360 (cversion, shift, ((1<<shift)-1)), file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
361 print("\t}", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
362 print("\treturn change_records_%s+index;" % cversion, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
363 print("}\n", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
364 print("static Py_UCS4 normalization_%s(Py_UCS4 n)" % cversion, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
365 print("{", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
366 print("\tswitch(n) {", file=fp)
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
367 for k, v in normalization:
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
368 print("\tcase %s: return 0x%s;" % (hex(k), v), file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
369 print("\tdefault: return 0;", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
370 print("\t}\n}\n", file=fp)
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
371
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
372 fp.close()
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
373
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
374 # --------------------------------------------------------------------
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
375 # unicode character type tables
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
376
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
377 def makeunicodetype(unicode, trace):
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
378
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
379 FILE = "Objects/unicodetype_db.h"
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
380
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
381 print("--- Preparing", FILE, "...")
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
382
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
383 # extract unicode types
14852
463faf01685e unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14849
diff changeset
384 dummy = (0, 0, 0, 0, 0, 0)
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
385 table = [dummy]
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
386 cache = {0: dummy}
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
387 index = [0] * len(unicode.chars)
56408
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
388 numeric = {}
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
389 spaces = []
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
390 linebreaks = []
74338
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
391 extra_casing = []
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
392
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
393 for char in unicode.chars:
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
394 record = unicode.table[char]
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
395 if record:
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
396 # extract database properties
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
397 category = record[2]
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
398 bidirectional = record[4]
42595
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 42427
diff changeset
399 properties = record[16]
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
400 flags = 0
49125
dc60293b29d5 Merged revisions 66362 via svnmerge from
Martin v. Löwis <martin@v.loewis.de>
parents: 48427
diff changeset
401 delta = True
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
402 if category in ["Lm", "Lt", "Lu", "Ll", "Lo"]:
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
403 flags |= ALPHA_MASK
74338
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
404 if "Lowercase" in properties:
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
405 flags |= LOWER_MASK
60086
bf3c48f2417c Merged revisions 79494,79496 via svnmerge from
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59720
diff changeset
406 if 'Line_Break' in properties or bidirectional == "B":
14852
463faf01685e unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14849
diff changeset
407 flags |= LINEBREAK_MASK
56408
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
408 linebreaks.append(char)
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
409 if category == "Zs" or bidirectional in ("WS", "B", "S"):
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
410 flags |= SPACE_MASK
56408
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
411 spaces.append(char)
14856
83aff1ea5b70 - don't set the titlecase flag for uppercase letters (sorry, tim)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14852
diff changeset
412 if category == "Lt":
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
413 flags |= TITLE_MASK
74338
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
414 if "Uppercase" in properties:
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
415 flags |= UPPER_MASK
52364
ca02bd8c96f2 Merged revisions 70610 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 51122
diff changeset
416 if char == ord(" ") or category[0] not in ("C", "Z"):
48427
4e6859c6d38c Issue #3282: str.isprintable() should return False for undefined Unicode characters.
Georg Brandl <georg@python.org>
parents: 47680
diff changeset
417 flags |= PRINTABLE_MASK
42595
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 42427
diff changeset
418 if "XID_Start" in properties:
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 42427
diff changeset
419 flags |= XID_START_MASK
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 42427
diff changeset
420 if "XID_Continue" in properties:
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 42427
diff changeset
421 flags |= XID_CONTINUE_MASK
74338
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
422 if "Cased" in properties:
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
423 flags |= CASED_MASK
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
424 if "Case_Ignorable" in properties:
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
425 flags |= CASE_IGNORABLE_MASK
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
426 sc = unicode.special_casing.get(char)
74405
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
427 cf = unicode.case_folding.get(char, [char])
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
428 if record[12]:
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
429 upper = int(record[12], 16)
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
430 else:
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
431 upper = char
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
432 if record[13]:
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
433 lower = int(record[13], 16)
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
434 else:
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
435 lower = char
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
436 if record[14]:
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
437 title = int(record[14], 16)
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
438 else:
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
439 title = upper
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
440 if sc is None and cf != [lower]:
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
441 sc = ([lower], [title], [upper])
74338
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
442 if sc is None:
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
443 if upper == lower == title:
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
444 upper = lower = title = 0
74423
03ea95e3b497 delta encoding of upper/lower/title makes a glorious return (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 74405
diff changeset
445 else:
03ea95e3b497 delta encoding of upper/lower/title makes a glorious return (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 74405
diff changeset
446 upper = upper - char
03ea95e3b497 delta encoding of upper/lower/title makes a glorious return (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 74405
diff changeset
447 lower = lower - char
03ea95e3b497 delta encoding of upper/lower/title makes a glorious return (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 74405
diff changeset
448 title = title - char
03ea95e3b497 delta encoding of upper/lower/title makes a glorious return (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 74405
diff changeset
449 assert (abs(upper) <= 2147483647 and
03ea95e3b497 delta encoding of upper/lower/title makes a glorious return (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 74405
diff changeset
450 abs(lower) <= 2147483647 and
03ea95e3b497 delta encoding of upper/lower/title makes a glorious return (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 74405
diff changeset
451 abs(title) <= 2147483647)
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
452 else:
74405
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
453 # This happens either when some character maps to more than one
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
454 # character in uppercase, lowercase, or titlecase or the
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
455 # casefolded version of the character is different from the
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
456 # lowercase. The extra characters are stored in a different
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
457 # array.
74338
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
458 flags |= EXTENDED_CASE_MASK
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
459 lower = len(extra_casing) | (len(sc[0]) << 24)
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
460 extra_casing.extend(sc[0])
74405
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
461 if cf != sc[0]:
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
462 lower |= len(cf) << 20
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
463 extra_casing.extend(cf)
74338
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
464 upper = len(extra_casing) | (len(sc[2]) << 24)
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
465 extra_casing.extend(sc[2])
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
466 # Title is probably equal to upper.
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
467 if sc[1] == sc[2]:
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
468 title = upper
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
469 else:
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
470 title = len(extra_casing) | (len(sc[1]) << 24)
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
471 extra_casing.extend(sc[1])
14852
463faf01685e unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14849
diff changeset
472 # decimal digit, integer digit
463faf01685e unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14849
diff changeset
473 decimal = 0
463faf01685e unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14849
diff changeset
474 if record[6]:
463faf01685e unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14849
diff changeset
475 flags |= DECIMAL_MASK
463faf01685e unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14849
diff changeset
476 decimal = int(record[6])
463faf01685e unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14849
diff changeset
477 digit = 0
463faf01685e unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14849
diff changeset
478 if record[7]:
463faf01685e unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14849
diff changeset
479 flags |= DIGIT_MASK
463faf01685e unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14849
diff changeset
480 digit = int(record[7])
56408
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
481 if record[8]:
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
482 flags |= NUMERIC_MASK
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
483 numeric.setdefault(record[8], []).append(char)
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
484 item = (
32197
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 26397
diff changeset
485 upper, lower, title, decimal, digit, flags
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
486 )
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
487 # add entry to index and item tables
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
488 i = cache.get(item)
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
489 if i is None:
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
490 cache[item] = i = len(table)
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
491 table.append(item)
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
492 index[char] = i
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
493
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
494 print(len(table), "unique character type entries")
56408
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
495 print(sum(map(len, numeric.values())), "numeric code points")
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
496 print(len(spaces), "whitespace code points")
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
497 print(len(linebreaks), "linebreak code points")
74338
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
498 print(len(extra_casing), "extended case array")
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
499
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
500 print("--- Writing", FILE, "...")
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
501
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
502 fp = open(FILE, "w")
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
503 print("/* this file was generated by %s %s */" % (SCRIPT, VERSION), file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
504 print(file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
505 print("/* a list of unique character type descriptors */", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
506 print("const _PyUnicode_TypeRecord _PyUnicode_TypeRecords[] = {", file=fp)
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
507 for item in table:
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
508 print(" {%d, %d, %d, %d, %d, %d}," % item, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
509 print("};", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
510 print(file=fp)
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
511
74338
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
512 print("/* extended case mappings */", file=fp)
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
513 print(file=fp)
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
514 print("const Py_UCS4 _PyUnicode_ExtendedCase[] = {", file=fp)
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
515 for c in extra_casing:
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
516 print(" %d," % c, file=fp)
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
517 print("};", file=fp)
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
518 print(file=fp)
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
519
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
520 # split decomposition index table
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
521 index1, index2, shift = splitbins(index, trace)
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
522
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
523 print("/* type indexes */", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
524 print("#define SHIFT", shift, file=fp)
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
525 Array("index1", index1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
526 Array("index2", index2).dump(fp, trace)
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
527
56408
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
528 # Generate code for _PyUnicode_ToNumeric()
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
529 numeric_items = sorted(numeric.items())
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
530 print('/* Returns the numeric value as double for Unicode characters', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
531 print(' * having this property, -1.0 otherwise.', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
532 print(' */', file=fp)
64146
8f97c64b2333 #5127: Even on narrow unicode builds, the C functions that access the Unicode
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 60086
diff changeset
533 print('double _PyUnicode_ToNumeric(Py_UCS4 ch)', file=fp)
56408
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
534 print('{', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
535 print(' switch (ch) {', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
536 for value, codepoints in numeric_items:
56508
e8e103488fb3 Merged revisions 75396 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 56408
diff changeset
537 # Turn text into float literals
e8e103488fb3 Merged revisions 75396 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 56408
diff changeset
538 parts = value.split('/')
e8e103488fb3 Merged revisions 75396 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 56408
diff changeset
539 parts = [repr(float(part)) for part in parts]
e8e103488fb3 Merged revisions 75396 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 56408
diff changeset
540 value = '/'.join(parts)
e8e103488fb3 Merged revisions 75396 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 56408
diff changeset
541
56408
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
542 codepoints.sort()
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
543 for codepoint in codepoints:
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
544 print(' case 0x%04X:' % (codepoint,), file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
545 print(' return (double) %s;' % (value,), file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
546 print(' }', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
547 print(' return -1.0;', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
548 print('}', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
549 print(file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
550
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
551 # Generate code for _PyUnicode_IsWhitespace()
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
552 print("/* Returns 1 for Unicode characters having the bidirectional", file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
553 print(" * type 'WS', 'B' or 'S' or the category 'Zs', 0 otherwise.", file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
554 print(" */", file=fp)
64146
8f97c64b2333 #5127: Even on narrow unicode builds, the C functions that access the Unicode
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 60086
diff changeset
555 print('int _PyUnicode_IsWhitespace(register const Py_UCS4 ch)', file=fp)
56408
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
556 print('{', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
557 print(' switch (ch) {', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
558
59720
8a28be45c242 Merged revisions 78982,78986 via svnmerge from
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59719
diff changeset
559 for codepoint in sorted(spaces):
56408
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
560 print(' case 0x%04X:' % (codepoint,), file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
561 print(' return 1;', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
562
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
563 print(' }', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
564 print(' return 0;', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
565 print('}', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
566 print(file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
567
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
568 # Generate code for _PyUnicode_IsLinebreak()
60086
bf3c48f2417c Merged revisions 79494,79496 via svnmerge from
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59720
diff changeset
569 print("/* Returns 1 for Unicode characters having the line break", file=fp)
bf3c48f2417c Merged revisions 79494,79496 via svnmerge from
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59720
diff changeset
570 print(" * property 'BK', 'CR', 'LF' or 'NL' or having bidirectional", file=fp)
bf3c48f2417c Merged revisions 79494,79496 via svnmerge from
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59720
diff changeset
571 print(" * type 'B', 0 otherwise.", file=fp)
56408
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
572 print(" */", file=fp)
64146
8f97c64b2333 #5127: Even on narrow unicode builds, the C functions that access the Unicode
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 60086
diff changeset
573 print('int _PyUnicode_IsLinebreak(register const Py_UCS4 ch)', file=fp)
56408
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
574 print('{', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
575 print(' switch (ch) {', file=fp)
59720
8a28be45c242 Merged revisions 78982,78986 via svnmerge from
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59719
diff changeset
576 for codepoint in sorted(linebreaks):
56408
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
577 print(' case 0x%04X:' % (codepoint,), file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
578 print(' return 1;', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
579
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
580 print(' }', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
581 print(' return 0;', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
582 print('}', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
583 print(file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
584
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
585 fp.close()
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
586
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
587 # --------------------------------------------------------------------
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
588 # unicode name database
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
589
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
590 def makeunicodename(unicode, trace):
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
591
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
592 FILE = "Modules/unicodename_db.h"
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
593
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
594 print("--- Preparing", FILE, "...")
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
595
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
596 # collect names
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
597 names = [None] * len(unicode.chars)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
598
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
599 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
600 record = unicode.table[char]
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
601 if record:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
602 name = record[1].strip()
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
603 if name and name[0] != "<":
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
604 names[char] = name + chr(0)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
605
47611
da47e7e135ae #2630: Implement PEP 3138.
Georg Brandl <georg@python.org>
parents: 47463
diff changeset
606 print(len(list(n for n in names if n is not None)), "distinct names")
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
607
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
608 # collect unique words from names (note that we differ between
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
609 # words inside a sentence, and words ending a sentence. the
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
610 # latter includes the trailing null byte.
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
611
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
612 words = {}
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
613 n = b = 0
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
614 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
615 name = names[char]
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
616 if name:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
617 w = name.split()
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
618 b = b + len(name)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
619 n = n + len(w)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
620 for w in w:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
621 l = words.get(w)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
622 if l:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
623 l.append(None)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
624 else:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
625 words[w] = [len(words)]
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
626
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
627 print(n, "words in text;", b, "bytes")
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
628
42595
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 42427
diff changeset
629 wordlist = list(words.items())
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
630
26394
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
631 # sort on falling frequency, then by name
51122
adbd77a90bde Merged revisions 69025 via svnmerge from
Mark Dickinson <dickinsm@gmail.com>
parents: 49125
diff changeset
632 def word_key(a):
42595
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 42427
diff changeset
633 aword, alist = a
51122
adbd77a90bde Merged revisions 69025 via svnmerge from
Mark Dickinson <dickinsm@gmail.com>
parents: 49125
diff changeset
634 return -len(alist), aword
adbd77a90bde Merged revisions 69025 via svnmerge from
Mark Dickinson <dickinsm@gmail.com>
parents: 49125
diff changeset
635 wordlist.sort(key=word_key)
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
636
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
637 # figure out how many phrasebook escapes we need
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
638 escapes = 0
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
639 while escapes * 256 < len(wordlist):
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
640 escapes = escapes + 1
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
641 print(escapes, "escapes")
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
642
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
643 short = 256 - escapes
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
644
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
645 assert short > 0
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
646
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
647 print(short, "short indexes in lexicon")
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
648
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
649 # statistics
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
650 n = 0
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
651 for i in range(short):
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
652 n = n + len(wordlist[i][1])
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
653 print(n, "short indexes in phrasebook")
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
654
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
655 # pick the most commonly used words, and sort the rest on falling
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
656 # length (to maximize overlap)
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
657
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
658 wordlist, wordtail = wordlist[:short], wordlist[short:]
45049
90e86ee3684b Convert some custom sort comparison functions to equivalent key functions.
Raymond Hettinger <python@rcn.com>
parents: 44160
diff changeset
659 wordtail.sort(key=lambda a: a[0], reverse=True)
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
660 wordlist.extend(wordtail)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
661
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
662 # generate lexicon from words
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
663
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
664 lexicon_offset = [0]
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
665 lexicon = ""
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
666 words = {}
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
667
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
668 # build a lexicon string
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
669 offset = 0
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
670 for w, x in wordlist:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
671 # encoding: bit 7 indicates last character in word (chr(128)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
672 # indicates the last character in an entire string)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
673 ww = w[:-1] + chr(ord(w[-1])+128)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
674 # reuse string tails, when possible
25219
8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113
Walter Dörwald <walter@livinglogic.de>
parents: 16091
diff changeset
675 o = lexicon.find(ww)
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
676 if o < 0:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
677 o = offset
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
678 lexicon = lexicon + ww
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
679 offset = offset + len(w)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
680 words[w] = len(lexicon_offset)
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
681 lexicon_offset.append(o)
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
682
42595
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 42427
diff changeset
683 lexicon = list(map(ord, lexicon))
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
684
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
685 # generate phrasebook from names and lexicon
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
686 phrasebook = [0]
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
687 phrasebook_offset = [0] * len(unicode.chars)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
688 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
689 name = names[char]
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
690 if name:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
691 w = name.split()
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
692 phrasebook_offset[char] = len(phrasebook)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
693 for w in w:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
694 i = words[w]
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
695 if i < short:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
696 phrasebook.append(i)
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
697 else:
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
698 # store as two bytes
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
699 phrasebook.append((i>>8) + short)
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
700 phrasebook.append(i&255)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
701
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
702 assert getsize(phrasebook) == 1
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
703
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
704 #
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
705 # unicode name hash table
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
706
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
707 # extract names
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
708 data = []
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
709 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
710 record = unicode.table[char]
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
711 if record:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
712 name = record[1].strip()
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
713 if name and name[0] != "<":
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
714 data.append((name, char))
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
715
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
716 # the magic number 47 was chosen to minimize the number of
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
717 # collisions on the current data set. if you like, change it
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
718 # and see what happens...
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
719
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
720 codehash = Hash("code", data, 47)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
721
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
722 print("--- Writing", FILE, "...")
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
723
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
724 fp = open(FILE, "w")
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
725 print("/* this file was generated by %s %s */" % (SCRIPT, VERSION), file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
726 print(file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
727 print("#define NAME_MAXLEN", 256, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
728 print(file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
729 print("/* lexicon */", file=fp)
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
730 Array("lexicon", lexicon).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
731 Array("lexicon_offset", lexicon_offset).dump(fp, trace)
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
732
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
733 # split decomposition index table
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
734 offset1, offset2, shift = splitbins(phrasebook_offset, trace)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
735
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
736 print("/* code->name phrasebook */", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
737 print("#define phrasebook_shift", shift, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
738 print("#define phrasebook_short", short, file=fp)
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
739
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
740 Array("phrasebook", phrasebook).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
741 Array("phrasebook_offset1", offset1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
742 Array("phrasebook_offset2", offset2).dump(fp, trace)
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
743
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
744 print("/* name->code dictionary */", file=fp)
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
745 codehash.dump(fp, trace)
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
746
73044
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
747 print(file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
748 print('static const unsigned int aliases_start = %#x;' %
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
749 NAME_ALIASES_START, file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
750 print('static const unsigned int aliases_end = %#x;' %
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
751 (NAME_ALIASES_START + len(unicode.aliases)), file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
752
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
753 print('static const unsigned int name_aliases[] = {', file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
754 for name, codepoint in unicode.aliases:
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
755 print(' 0x%04X,' % codepoint, file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
756 print('};', file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
757
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
758 # In Unicode 6.0.0, the sequences contain at most 4 BMP chars,
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
759 # so we are using Py_UCS2 seq[4]. This needs to be updated if longer
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
760 # sequences or sequences with non-BMP chars are added.
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
761 # unicodedata_lookup should be adapted too.
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
762 print(dedent("""
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
763 typedef struct NamedSequence {
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
764 int seqlen;
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
765 Py_UCS2 seq[4];
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
766 } named_sequence;
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
767 """), file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
768
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
769 print('static const unsigned int named_sequences_start = %#x;' %
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
770 NAMED_SEQUENCES_START, file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
771 print('static const unsigned int named_sequences_end = %#x;' %
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
772 (NAMED_SEQUENCES_START + len(unicode.named_sequences)), file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
773
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
774 print('static const named_sequence named_sequences[] = {', file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
775 for name, sequence in unicode.named_sequences:
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
776 seq_str = ', '.join('0x%04X' % cp for cp in sequence)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
777 print(' {%d, {%s}},' % (len(sequence), seq_str), file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
778 print('};', file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
779
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
780 fp.close()
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
781
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
782
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
783 def merge_old_version(version, new, old):
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
784 # Changes to exclusion file not implemented yet
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
785 if old.exclusions != new.exclusions:
42715
e4ade16a694f Convert raise statements in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42595
diff changeset
786 raise NotImplementedError("exclusions differ")
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
787
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
788 # In these change records, 0xFF means "no change"
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
789 bidir_changes = [0xFF]*0x110000
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
790 category_changes = [0xFF]*0x110000
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
791 decimal_changes = [0xFF]*0x110000
49125
dc60293b29d5 Merged revisions 66362 via svnmerge from
Martin v. Löwis <martin@v.loewis.de>
parents: 48427
diff changeset
792 mirrored_changes = [0xFF]*0x110000
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
793 # In numeric data, 0 means "no change",
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
794 # -1 means "did not have a numeric value
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
795 numeric_changes = [0] * 0x110000
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
796 # normalization_changes is a list of key-value pairs
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
797 normalization_changes = []
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
798 for i in range(0x110000):
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
799 if new.table[i] is None:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
800 # Characters unassigned in the new version ought to
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
801 # be unassigned in the old one
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
802 assert old.table[i] is None
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
803 continue
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
804 # check characters unassigned in the old version
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
805 if old.table[i] is None:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
806 # category 0 is "unassigned"
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
807 category_changes[i] = 0
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
808 continue
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
809 # check characters that differ
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
810 if old.table[i] != new.table[i]:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
811 for k in range(len(old.table[i])):
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
812 if old.table[i][k] != new.table[i][k]:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
813 value = old.table[i][k]
73044
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
814 if k == 1 and i in PUA_15:
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
815 # the name is not set in the old.table, but in the
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
816 # new.table we are using it for aliases and named seq
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
817 assert value == ''
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
818 elif k == 2:
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
819 #print "CATEGORY",hex(i), old.table[i][k], new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
820 category_changes[i] = CATEGORY_NAMES.index(value)
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
821 elif k == 4:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
822 #print "BIDIR",hex(i), old.table[i][k], new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
823 bidir_changes[i] = BIDIRECTIONAL_NAMES.index(value)
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
824 elif k == 5:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
825 #print "DECOMP",hex(i), old.table[i][k], new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
826 # We assume that all normalization changes are in 1:1 mappings
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
827 assert " " not in value
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
828 normalization_changes.append((i, value))
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
829 elif k == 6:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
830 #print "DECIMAL",hex(i), old.table[i][k], new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
831 # we only support changes where the old value is a single digit
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
832 assert value in "0123456789"
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
833 decimal_changes[i] = int(value)
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
834 elif k == 8:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
835 # print "NUMERIC",hex(i), `old.table[i][k]`, new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
836 # Since 0 encodes "no change", the old value is better not 0
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
837 if not value:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
838 numeric_changes[i] = -1
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
839 else:
56408
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
840 numeric_changes[i] = float(value)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
841 assert numeric_changes[i] not in (0, -1)
49125
dc60293b29d5 Merged revisions 66362 via svnmerge from
Martin v. Löwis <martin@v.loewis.de>
parents: 48427
diff changeset
842 elif k == 9:
dc60293b29d5 Merged revisions 66362 via svnmerge from
Martin v. Löwis <martin@v.loewis.de>
parents: 48427
diff changeset
843 if value == 'Y':
dc60293b29d5 Merged revisions 66362 via svnmerge from
Martin v. Löwis <martin@v.loewis.de>
parents: 48427
diff changeset
844 mirrored_changes[i] = '1'
dc60293b29d5 Merged revisions 66362 via svnmerge from
Martin v. Löwis <martin@v.loewis.de>
parents: 48427
diff changeset
845 else:
dc60293b29d5 Merged revisions 66362 via svnmerge from
Martin v. Löwis <martin@v.loewis.de>
parents: 48427
diff changeset
846 mirrored_changes[i] = '0'
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
847 elif k == 11:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
848 # change to ISO comment, ignore
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
849 pass
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
850 elif k == 12:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
851 # change to simple uppercase mapping; ignore
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
852 pass
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
853 elif k == 13:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
854 # change to simple lowercase mapping; ignore
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
855 pass
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
856 elif k == 14:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
857 # change to simple titlecase mapping; ignore
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
858 pass
42595
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 42427
diff changeset
859 elif k == 16:
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 42427
diff changeset
860 # derived property changes; not yet
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 42427
diff changeset
861 pass
65217
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
862 elif k == 17:
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
863 # normalization quickchecks are not performed
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
864 # for older versions
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
865 pass
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
866 else:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
867 class Difference(Exception):pass
42715
e4ade16a694f Convert raise statements in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42595
diff changeset
868 raise Difference(hex(i), k, old.table[i], new.table[i])
42595
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 42427
diff changeset
869 new.changed.append((version, list(zip(bidir_changes, category_changes,
49125
dc60293b29d5 Merged revisions 66362 via svnmerge from
Martin v. Löwis <martin@v.loewis.de>
parents: 48427
diff changeset
870 decimal_changes, mirrored_changes,
dc60293b29d5 Merged revisions 66362 via svnmerge from
Martin v. Löwis <martin@v.loewis.de>
parents: 48427
diff changeset
871 numeric_changes)),
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
872 normalization_changes))
36937
df43c04d2554 Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 36924
diff changeset
873
65217
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
874 def open_data(template, version):
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
875 local = template % ('-'+version,)
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
876 if not os.path.exists(local):
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
877 import urllib.request
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
878 if version == '3.2.0':
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
879 # irregular url structure
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
880 url = 'http://www.unicode.org/Public/3.2-Update/' + local
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
881 else:
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
882 url = ('http://www.unicode.org/Public/%s/ucd/'+template) % (version, '')
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
883 urllib.request.urlretrieve(url, filename=local)
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
884 if local.endswith('.txt'):
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
885 return open(local, encoding='utf-8')
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
886 else:
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
887 # Unihan.zip
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
888 return open(local, 'rb')
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
889
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
890 # --------------------------------------------------------------------
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
891 # the following support code is taken from the unidb utilities
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
892 # Copyright (c) 1999-2000 by Secret Labs AB
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
893
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
894 # load a unicode-data file from disk
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
895
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
896 class UnicodeData:
42595
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 42427
diff changeset
897 # Record structure:
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 42427
diff changeset
898 # [ID, name, category, combining, bidi, decomp, (6)
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 42427
diff changeset
899 # decimal, digit, numeric, bidi-mirrored, Unicode-1-name, (11)
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 42427
diff changeset
900 # ISO-comment, uppercase, lowercase, titlecase, ea-width, (16)
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 42427
diff changeset
901 # derived-props] (17)
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
902
65217
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
903 def __init__(self, version,
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
904 linebreakprops=False,
66392
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 65217
diff changeset
905 expand=1,
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 65217
diff changeset
906 cjk_check=True):
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
907 self.changed = []
25852
1f5f6481886d Update to Unicode 3.2 database.
Martin v. Löwis <martin@v.loewis.de>
parents: 25219
diff changeset
908 table = [None] * 0x110000
72547
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
909 with open_data(UNICODE_DATA, version) as file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
910 while 1:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
911 s = file.readline()
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
912 if not s:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
913 break
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
914 s = s.strip().split(";")
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
915 char = int(s[0], 16)
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
916 table[char] = s
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
917
66392
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 65217
diff changeset
918 cjk_ranges_found = []
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 65217
diff changeset
919
26394
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
920 # expand first-last ranges
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
921 if expand:
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
922 field = None
26394
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
923 for i in range(0, 0x110000):
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
924 s = table[i]
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
925 if s:
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
926 if s[1][-6:] == "First>":
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
927 s[1] = ""
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
928 field = s
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
929 elif s[1][-5:] == "Last>":
66392
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 65217
diff changeset
930 if s[1].startswith("<CJK Ideograph"):
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 65217
diff changeset
931 cjk_ranges_found.append((field[0],
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 65217
diff changeset
932 s[0]))
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
933 s[1] = ""
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
934 field = None
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
935 elif field:
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
936 f2 = field[:]
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
937 f2[0] = "%X" % i
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
938 table[i] = f2
66392
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 65217
diff changeset
939 if cjk_check and cjk_ranges != cjk_ranges_found:
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 65217
diff changeset
940 raise ValueError("CJK ranges deviate: have %r" % cjk_ranges_found)
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
941
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
942 # public attributes
65217
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
943 self.filename = UNICODE_DATA % ''
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
944 self.table = table
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 45049
diff changeset
945 self.chars = list(range(0x110000)) # unicode 3.2
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
946
73044
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
947 # check for name aliases and named sequences, see #12753
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
948 # aliases and named sequences are not in 3.2.0
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
949 if version != '3.2.0':
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
950 self.aliases = []
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
951 # store aliases in the Private Use Area 15, in range U+F0000..U+F00FF,
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
952 # in order to take advantage of the compression and lookup
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
953 # algorithms used for the other characters
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
954 pua_index = NAME_ALIASES_START
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
955 with open_data(NAME_ALIASES, version) as file:
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
956 for s in file:
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
957 s = s.strip()
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
958 if not s or s.startswith('#'):
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
959 continue
75107
cd21e6d30a1a update to Unicode 6.1
Benjamin Peterson <benjamin@python.org>
parents: 74423
diff changeset
960 char, name, abbrev = s.split(';')
73044
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
961 char = int(char, 16)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
962 self.aliases.append((name, char))
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
963 # also store the name in the PUA 1
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
964 self.table[pua_index][1] = name
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
965 pua_index += 1
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
966 assert pua_index - NAME_ALIASES_START == len(self.aliases)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
967
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
968 self.named_sequences = []
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
969 # store named seqences in the PUA 1, in range U+F0100..,
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
970 # in order to take advantage of the compression and lookup
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
971 # algorithms used for the other characters.
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
972
75107
cd21e6d30a1a update to Unicode 6.1
Benjamin Peterson <benjamin@python.org>
parents: 74423
diff changeset
973 assert pua_index < NAMED_SEQUENCES_START
73044
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
974 pua_index = NAMED_SEQUENCES_START
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
975 with open_data(NAMED_SEQUENCES, version) as file:
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
976 for s in file:
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
977 s = s.strip()
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
978 if not s or s.startswith('#'):
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
979 continue
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
980 name, chars = s.split(';')
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
981 chars = tuple(int(char, 16) for char in chars.split())
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
982 # check that the structure defined in makeunicodename is OK
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
983 assert 2 <= len(chars) <= 4, "change the Py_UCS2 array size"
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
984 assert all(c <= 0xFFFF for c in chars), ("use Py_UCS4 in "
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
985 "the NamedSequence struct and in unicodedata_lookup")
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
986 self.named_sequences.append((name, chars))
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
987 # also store these in the PUA 1
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
988 self.table[pua_index][1] = name
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
989 pua_index += 1
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
990 assert pua_index - NAMED_SEQUENCES_START == len(self.named_sequences)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 72547
diff changeset
991
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
992 self.exclusions = {}
72547
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
993 with open_data(COMPOSITION_EXCLUSIONS, version) as file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
994 for s in file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
995 s = s.strip()
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
996 if not s:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
997 continue
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
998 if s[0] == '#':
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
999 continue
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1000 char = int(s.split()[0],16)
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1001 self.exclusions[char] = 1
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
1002
32197
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 26397
diff changeset
1003 widths = [None] * 0x110000
72547
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1004 with open_data(EASTASIAN_WIDTH, version) as file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1005 for s in file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1006 s = s.strip()
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1007 if not s:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1008 continue
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1009 if s[0] == '#':
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1010 continue
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1011 s = s.split()[0].split(';')
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1012 if '..' in s[0]:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1013 first, last = [int(c, 16) for c in s[0].split('..')]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1014 chars = list(range(first, last+1))
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1015 else:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1016 chars = [int(s[0], 16)]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1017 for char in chars:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1018 widths[char] = s[1]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1019
32197
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 26397
diff changeset
1020 for i in range(0, 0x110000):
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 26397
diff changeset
1021 if table[i] is not None:
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 26397
diff changeset
1022 table[i].append(widths[i])
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 26397
diff changeset
1023
42595
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 42427
diff changeset
1024 for i in range(0, 0x110000):
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 42427
diff changeset
1025 if table[i] is not None:
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 42427
diff changeset
1026 table[i].append(set())
72547
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1027
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1028 with open_data(DERIVED_CORE_PROPERTIES, version) as file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1029 for s in file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1030 s = s.split('#', 1)[0].strip()
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1031 if not s:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1032 continue
42595
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 42427
diff changeset
1033
72547
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1034 r, p = s.split(";")
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1035 r = r.strip()
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1036 p = p.strip()
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1037 if ".." in r:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1038 first, last = [int(c, 16) for c in r.split('..')]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1039 chars = list(range(first, last+1))
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1040 else:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1041 chars = [int(r, 16)]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1042 for char in chars:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1043 if table[char]:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1044 # Some properties (e.g. Default_Ignorable_Code_Point)
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1045 # apply to unassigned code points; ignore them
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1046 table[char][-1].add(p)
42595
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 42427
diff changeset
1047
72547
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1048 with open_data(LINE_BREAK, version) as file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1049 for s in file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1050 s = s.partition('#')[0]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1051 s = [i.strip() for i in s.split(';')]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1052 if len(s) < 2 or s[1] not in MANDATORY_LINE_BREAKS:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1053 continue
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1054 if '..' not in s[0]:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1055 first = last = int(s[0], 16)
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1056 else:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1057 first, last = [int(c, 16) for c in s[0].split('..')]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1058 for char in range(first, last+1):
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1059 table[char][-1].add('Line_Break')
60086
bf3c48f2417c Merged revisions 79494,79496 via svnmerge from
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59720
diff changeset
1060
65217
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
1061 # We only want the quickcheck properties
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
1062 # Format: NF?_QC; Y(es)/N(o)/M(aybe)
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
1063 # Yes is the default, hence only N and M occur
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
1064 # In 3.2.0, the format was different (NF?_NO)
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
1065 # The parsing will incorrectly determine these as
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
1066 # "yes", however, unicodedata.c will not perform quickchecks
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
1067 # for older versions, and no delta records will be created.
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
1068 quickchecks = [0] * 0x110000
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
1069 qc_order = 'NFD_QC NFKD_QC NFC_QC NFKC_QC'.split()
72547
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1070 with open_data(DERIVEDNORMALIZATION_PROPS, version) as file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1071 for s in file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1072 if '#' in s:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1073 s = s[:s.index('#')]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1074 s = [i.strip() for i in s.split(';')]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1075 if len(s) < 2 or s[1] not in qc_order:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1076 continue
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1077 quickcheck = 'MN'.index(s[2]) + 1 # Maybe or No
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1078 quickcheck_shift = qc_order.index(s[1])*2
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1079 quickcheck <<= quickcheck_shift
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1080 if '..' not in s[0]:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1081 first = last = int(s[0], 16)
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1082 else:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1083 first, last = [int(c, 16) for c in s[0].split('..')]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1084 for char in range(first, last+1):
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1085 assert not (quickchecks[char]>>quickcheck_shift)&3
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1086 quickchecks[char] |= quickcheck
65217
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
1087 for i in range(0, 0x110000):
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
1088 if table[i] is not None:
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
1089 table[i].append(quickchecks[i])
53554
423f8f9b07b1 Merged revisions 72054 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 53402
diff changeset
1090
72547
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1091 with open_data(UNIHAN, version) as file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1092 zip = zipfile.ZipFile(file)
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1093 if version == '3.2.0':
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1094 data = zip.open('Unihan-3.2.0.txt').read()
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1095 else:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 68586
diff changeset
1096 data = zip.open('Unihan_NumericValues.txt').read()
65217
cbde81ef4af7 Upgrade to Unicode 6.0.0.
Martin v. Löwis <martin@v.loewis.de>
parents: 64687
diff changeset
1097 for line in data.decode("utf-8").splitlines():
56408
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
1098 if not line.startswith('U+'):
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
1099 continue
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
1100 code, tag, value = line.split(None, 3)[:3]
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
1101 if tag not in ('kAccountingNumeric', 'kPrimaryNumeric',
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
1102 'kOtherNumeric'):
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
1103 continue
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
1104 value = value.strip().replace(',', '')
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
1105 i = int(code[2:], 16)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
1106 # Patch the numeric field
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
1107 if table[i] is not None:
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53554
diff changeset
1108 table[i][8] = value
74338
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
1109 sc = self.special_casing = {}
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
1110 with open_data(SPECIAL_CASING, version) as file:
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
1111 for s in file:
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
1112 s = s[:-1].split('#', 1)[0]
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
1113 if not s:
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
1114 continue
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
1115 data = s.split("; ")
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
1116 if data[4]:
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
1117 # We ignore all conditionals (since they depend on
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
1118 # languages) except for one, which is hardcoded. See
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
1119 # handle_capital_sigma in unicodeobject.c.
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
1120 continue
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
1121 c = int(data[0], 16)
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
1122 lower = [int(char, 16) for char in data[1].split()]
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
1123 title = [int(char, 16) for char in data[2].split()]
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
1124 upper = [int(char, 16) for char in data[3].split()]
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736)
Benjamin Peterson <benjamin@python.org>
parents: 73044
diff changeset
1125 sc[c] = (lower, title, upper)
74405
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
1126 cf = self.case_folding = {}
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
1127 if version != '3.2.0':
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
1128 with open_data(CASE_FOLDING, version) as file:
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
1129 for s in file:
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
1130 s = s[:-1].split('#', 1)[0]
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
1131 if not s:
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
1132 continue
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
1133 data = s.split("; ")
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
1134 if data[1] in "CF":
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
1135 c = int(data[0], 16)
d4669f43d05f add str.casefold() (closes #13752)
Benjamin Peterson <benjamin@python.org>
parents: 74338
diff changeset
1136 cf[c] = [int(char, 16) for char in data[2].split()]
53554
423f8f9b07b1 Merged revisions 72054 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 53402
diff changeset
1137
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1138 def uselatin1(self):
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1139 # restrict character range to ISO Latin 1
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 45049
diff changeset
1140 self.chars = list(range(256))
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1141
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1142 # hash table tools
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1143
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1144 # this is a straight-forward reimplementation of Python's built-in
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1145 # dictionary type, using a static data structure, and a custom string
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1146 # hash algorithm.
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1147
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1148 def myhash(s, magic):
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1149 h = 0
25219
8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113
Walter Dörwald <walter@livinglogic.de>
parents: 16091
diff changeset
1150 for c in map(ord, s.upper()):
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1151 h = (h * magic) + c
42084
229c28cb9afb Merged revisions 55817-55961 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 41642
diff changeset
1152 ix = h & 0xff000000
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1153 if ix:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1154 h = (h ^ ((ix>>24) & 0xff)) & 0x00ffffff
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1155 return h
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1156
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1157 SIZES = [
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1158 (4,3), (8,3), (16,3), (32,5), (64,3), (128,3), (256,29), (512,17),
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1159 (1024,9), (2048,5), (4096,83), (8192,27), (16384,43), (32768,3),
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1160 (65536,45), (131072,9), (262144,39), (524288,39), (1048576,9),
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1161 (2097152,5), (4194304,3), (8388608,33), (16777216,27)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1162 ]
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1163
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1164 class Hash:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1165 def __init__(self, name, data, magic):
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1166 # turn a (key, value) list into a static hash table structure
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1167
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1168 # determine table size
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1169 for size, poly in SIZES:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1170 if size > len(data):
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1171 poly = size + poly
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1172 break
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1173 else:
68585
bf94b6a73fc8 #11565: Fix several typos. Patch by Piotr Kasprzyk.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 53554
diff changeset
1174 raise AssertionError("ran out of polynomials")
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1175
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
1176 print(size, "slots in hash table")
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1177
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1178 table = [None] * size
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1179
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1180 mask = size-1
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1181
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1182 n = 0
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1183
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1184 hash = myhash
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1185
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1186 # initialize hash table
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1187 for key, value in data:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1188 h = hash(key, magic)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1189 i = (~h) & mask
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1190 v = table[i]
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1191 if v is None:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1192 table[i] = value
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1193 continue
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1194 incr = (h ^ (h >> 3)) & mask;
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1195 if not incr:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1196 incr = mask
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1197 while 1:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1198 n = n + 1
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1199 i = (i + incr) & mask
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1200 v = table[i]
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1201 if v is None:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1202 table[i] = value
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1203 break
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1204 incr = incr << 1
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1205 if incr > mask:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1206 incr = incr ^ poly
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1207
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
1208 print(n, "collisions")
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1209 self.collisions = n
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1210
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1211 for i in range(len(table)):
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1212 if table[i] is None:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1213 table[i] = 0
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1214
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1215 self.data = Array(name + "_hash", table)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1216 self.magic = magic
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1217 self.name = name
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1218 self.size = size
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1219 self.poly = poly
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1220
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
1221 def dump(self, file, trace):
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1222 # write data to file, as a C array
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
1223 self.data.dump(file, trace)
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1224 file.write("#define %s_magic %d\n" % (self.name, self.magic))
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1225 file.write("#define %s_size %d\n" % (self.name, self.size))
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1226 file.write("#define %s_poly %d\n" % (self.name, self.poly))
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1227
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1228 # stuff to deal with arrays of unsigned integers
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1229
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1230 class Array:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1231
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1232 def __init__(self, name, data):
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1233 self.name = name
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1234 self.data = data
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1235
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
1236 def dump(self, file, trace=0):
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1237 # write data to file, as a C array
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1238 size = getsize(self.data)
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
1239 if trace:
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
1240 print(self.name+":", size*len(self.data), "bytes", file=sys.stderr)
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1241 file.write("static ")
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1242 if size == 1:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1243 file.write("unsigned char")
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1244 elif size == 2:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1245 file.write("unsigned short")
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1246 else:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1247 file.write("unsigned int")
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1248 file.write(" " + self.name + "[] = {\n")
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1249 if self.data:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1250 s = " "
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1251 for item in self.data:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1252 i = str(item) + ", "
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1253 if len(s) + len(i) > 78:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1254 file.write(s + "\n")
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1255 s = " " + i
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1256 else:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1257 s = s + i
25219
8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113
Walter Dörwald <walter@livinglogic.de>
parents: 16091
diff changeset
1258 if s.strip():
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1259 file.write(s + "\n")
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1260 file.write("};\n\n")
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1261
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1262 def getsize(data):
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1263 # return smallest possible integer size for the given array
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1264 maxdata = max(data)
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1265 if maxdata < 256:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1266 return 1
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1267 elif maxdata < 65536:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1268 return 2
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1269 else:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1270 return 4
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1271
14834
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1272 def splitbins(t, trace=0):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1273 """t, trace=0 -> (t1, t2, shift). Split a table to save space.
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1274
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1275 t is a sequence of ints. This function can be useful to save space if
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1276 many of the ints are the same. t1 and t2 are lists of ints, and shift
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1277 is an int, chosen to minimize the combined size of t1 and t2 (in C
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1278 code), and where for each i in range(len(t)),
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1279 t[i] == t2[(t1[i >> shift] << shift) + (i & mask)]
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1280 where mask is a bitmask isolating the last "shift" bits.
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1281
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
1282 If optional arg trace is non-zero (default zero), progress info
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
1283 is printed to sys.stderr. The higher the value, the more info
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
1284 you'll get.
14834
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1285 """
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1286
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1287 if trace:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1288 def dump(t1, t2, shift, bytes):
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
1289 print("%d+%d bins at shift %d; %d bytes" % (
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
1290 len(t1), len(t2), shift, bytes), file=sys.stderr)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
1291 print("Size of original table:", len(t)*getsize(t), \
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
1292 "bytes", file=sys.stderr)
14834
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1293 n = len(t)-1 # last valid index
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1294 maxshift = 0 # the most we can shift n and still have something left
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1295 if n > 0:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1296 while n >> 1:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1297 n >>= 1
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1298 maxshift += 1
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1299 del n
44160
9d152530910a Removed PyInt_GetMax and sys.maxint
Christian Heimes <christian@cheimes.de>
parents: 42715
diff changeset
1300 bytes = sys.maxsize # smallest total size so far
14834
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1301 t = tuple(t) # so slices can be dict keys
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1302 for shift in range(maxshift + 1):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1303 t1 = []
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1304 t2 = []
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1305 size = 2**shift
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1306 bincache = {}
14834
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1307 for i in range(0, len(t), size):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1308 bin = t[i:i+size]
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1309 index = bincache.get(bin)
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1310 if index is None:
14834
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1311 index = len(t2)
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1312 bincache[bin] = index
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1313 t2.extend(bin)
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1314 t1.append(index >> shift)
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1315 # determine memory size
14834
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1316 b = len(t1)*getsize(t1) + len(t2)*getsize(t2)
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
1317 if trace > 1:
14834
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1318 dump(t1, t2, shift, b)
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1319 if b < bytes:
14834
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1320 best = t1, t2, shift
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1321 bytes = b
14834
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1322 t1, t2, shift = best
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1323 if trace:
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42084
diff changeset
1324 print("Best:", end=' ', file=sys.stderr)
14834
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1325 dump(t1, t2, shift, bytes)
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1326 if __debug__:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1327 # exhaustively verify that the decomposition is correct
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1328 mask = ~((~0) << shift) # i.e., low-bit mask of shift bits
41642
a6eb6acfe04a Merged revisions 55007-55179 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 36940
diff changeset
1329 for i in range(len(t)):
14834
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1330 assert t[i] == t2[(t1[i >> shift] << shift) + (i & mask)]
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1331 return best
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1332
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1333 if __name__ == "__main__":
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
1334 maketables(1)