Mercurial > cpython-lru_cache-weakref

annotate Tools/unicode/makeunicodedata.py @ 80661:94de77bd0b4b 2.7

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Issue #16194: document imp.load_dynamic problems when read several extension modules from the same binary object file.
author Andrew Svetlov <andrew.svetlov@gmail.com>
date Thu, 29 Nov 2012 19:51:28 +0200
parents 66e757d8ea93
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1 #
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
2 # (re)generate unicode property and type databases
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
3 #
26397
b27c1cad77ea Add unidata_version. Bump generator version number.
Martin v. Löwis <martin@v.loewis.de>
parents: 26394
diff changeset
4 # this script converts a unicode 3.2 database file to
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
5 # Modules/unicodedata_db.h, Modules/unicodename_db.h,
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
6 # and Objects/unicodetype_db.h
14835
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
7 #
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
8 # history:
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
9 # 2000-09-24 fl created (based on bits and pieces from unidb)
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
10 # 2000-09-25 fl merged tim's splitbin fixes, separate decomposition table
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
11 # 2000-09-25 fl added character type table
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
12 # 2000-09-26 fl added LINEBREAK, DECIMAL, and DIGIT flags/fields (2.0)
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
13 # 2000-11-03 fl expand first/last ranges
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
14 # 2001-01-19 fl added character name tables (2.1)
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
15 # 2001-01-21 fl added decomp compression; dynamic phrasebook threshold
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
16 # 2002-09-11 wd use string methods
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
17 # 2002-10-18 mvl update to Unicode 3.2
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
18 # 2002-10-22 mvl generate NFC tables
26394
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
19 # 2002-11-24 mvl expand all ranges, sort names version-independently
26397
b27c1cad77ea Add unidata_version. Bump generator version number.
Martin v. Löwis <martin@v.loewis.de>
parents: 26394
diff changeset
20 # 2002-11-25 mvl add UNIDATA_VERSION
32197
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 26397
diff changeset
21 # 2004-05-29 perky add east asian width information
36940
e4465d2737c4 Add changelog entry.
Martin v. Löwis <martin@v.loewis.de>
parents: 36937
diff changeset
22 # 2006-03-10 mvl update to Unicode 4.1; add UCD 3.2 delta
14835
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
23 #
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
24 # written by Fredrik Lundh (fredrik@pythonware.com)
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
25 #
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
26
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
27 import sys
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
28
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
29 SCRIPT = sys.argv[0]
49124
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 47678
diff changeset
30 VERSION = "2.6"
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
31
26397
b27c1cad77ea Add unidata_version. Bump generator version number.
Martin v. Löwis <martin@v.loewis.de>
parents: 26394
diff changeset
32 # The Unicode Database
59693
344cb6a4ef7d Issue #8024: Update the Unicode database to 5.2
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59640
diff changeset
33 UNIDATA_VERSION = "5.2.0"
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
34 UNICODE_DATA = "UnicodeData%s.txt"
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
35 COMPOSITION_EXCLUSIONS = "CompositionExclusions%s.txt"
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
36 EASTASIAN_WIDTH = "EastAsianWidth%s.txt"
56406
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
37 UNIHAN = "Unihan%s.txt"
53553
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 53400
diff changeset
38 DERIVEDNORMALIZATION_PROPS = "DerivedNormalizationProps%s.txt"
60074
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
39 LINE_BREAK = "LineBreak%s.txt"
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
40
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
41 old_versions = ["3.2.0"]
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
42
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
43 CATEGORY_NAMES = [ "Cn", "Lu", "Ll", "Lt", "Mn", "Mc", "Me", "Nd",
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
44 "Nl", "No", "Zs", "Zl", "Zp", "Cc", "Cf", "Cs", "Co", "Cn", "Lm",
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
45 "Lo", "Pc", "Pd", "Ps", "Pe", "Pi", "Pf", "Po", "Sm", "Sc", "Sk",
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
46 "So" ]
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
47
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
48 BIDIRECTIONAL_NAMES = [ "", "L", "LRE", "LRO", "R", "AL", "RLE", "RLO",
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
49 "PDF", "EN", "ES", "ET", "AN", "CS", "NSM", "BN", "B", "S", "WS",
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
50 "ON" ]
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
51
32951
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32197
diff changeset
52 EASTASIANWIDTH_NAMES = [ "F", "H", "W", "Na", "A", "N" ]
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32197
diff changeset
53
60074
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
54 MANDATORY_LINE_BREAKS = [ "BK", "CR", "LF", "NL" ]
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
55
14852
463faf01685e unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14849
diff changeset
56 # note: should match definitions in Objects/unicodectype.c
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
57 ALPHA_MASK = 0x01
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
58 DECIMAL_MASK = 0x02
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
59 DIGIT_MASK = 0x04
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
60 LOWER_MASK = 0x08
14852
463faf01685e unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14849
diff changeset
61 LINEBREAK_MASK = 0x10
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
62 SPACE_MASK = 0x20
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
63 TITLE_MASK = 0x40
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
64 UPPER_MASK = 0x80
49124
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 47678
diff changeset
65 NODELTA_MASK = 0x100
56406
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
66 NUMERIC_MASK = 0x200
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
67
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
68 def maketables(trace=0):
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
69
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
70 print "--- Reading", UNICODE_DATA % "", "..."
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
71
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
72 version = ""
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
73 unicode = UnicodeData(UNICODE_DATA % version,
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
74 COMPOSITION_EXCLUSIONS % version,
53553
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 53400
diff changeset
75 EASTASIAN_WIDTH % version,
56406
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
76 UNIHAN % version,
60074
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
77 DERIVEDNORMALIZATION_PROPS % version,
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
78 LINE_BREAK % version)
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
79
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
80 print len(filter(None, unicode.table)), "characters"
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
81
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
82 for version in old_versions:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
83 print "--- Reading", UNICODE_DATA % ("-"+version), "..."
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
84 old_unicode = UnicodeData(UNICODE_DATA % ("-"+version),
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
85 COMPOSITION_EXCLUSIONS % ("-"+version),
56406
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
86 EASTASIAN_WIDTH % ("-"+version),
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
87 UNIHAN % ("-"+version))
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
88 print len(filter(None, old_unicode.table)), "characters"
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
89 merge_old_version(version, unicode, old_unicode)
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
90
16091
45c065668ea5 Unicode nits: Don't include unicodedatabase.h no more. And make sure
Fredrik Lundh <fredrik@pythonware.com>
parents: 16090
diff changeset
91 makeunicodename(unicode, trace)
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
92 makeunicodedata(unicode, trace)
16091
45c065668ea5 Unicode nits: Don't include unicodedatabase.h no more. And make sure
Fredrik Lundh <fredrik@pythonware.com>
parents: 16090
diff changeset
93 makeunicodetype(unicode, trace)
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
94
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
95 # --------------------------------------------------------------------
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
96 # unicode character properties
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
97
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
98 def makeunicodedata(unicode, trace):
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
99
53553
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 53400
diff changeset
100 dummy = (0, 0, 0, 0, 0, 0)
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
101 table = [dummy]
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
102 cache = {0: dummy}
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
103 index = [0] * len(unicode.chars)
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
104
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
105 FILE = "Modules/unicodedata_db.h"
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
106
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
107 print "--- Preparing", FILE, "..."
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
108
14835
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
109 # 1) database properties
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
110
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
111 for char in unicode.chars:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
112 record = unicode.table[char]
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
113 if record:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
114 # extract database properties
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
115 category = CATEGORY_NAMES.index(record[2])
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
116 combining = int(record[3])
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
117 bidirectional = BIDIRECTIONAL_NAMES.index(record[4])
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
118 mirrored = record[9] == "Y"
32951
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32197
diff changeset
119 eastasianwidth = EASTASIANWIDTH_NAMES.index(record[15])
60074
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
120 normalizationquickcheck = record[17]
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
121 item = (
53553
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 53400
diff changeset
122 category, combining, bidirectional, mirrored, eastasianwidth,
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 53400
diff changeset
123 normalizationquickcheck
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
124 )
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
125 # add entry to index and item tables
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
126 i = cache.get(item)
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
127 if i is None:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
128 cache[item] = i = len(table)
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
129 table.append(item)
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
130 index[char] = i
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
131
14835
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
132 # 2) decomposition data
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
133
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
134 decomp_data = [0]
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
135 decomp_prefix = [""]
14835
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
136 decomp_index = [0] * len(unicode.chars)
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
137 decomp_size = 0
14835
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
138
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
139 comp_pairs = []
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
140 comp_first = [None] * len(unicode.chars)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
141 comp_last = [None] * len(unicode.chars)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
142
14835
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
143 for char in unicode.chars:
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
144 record = unicode.table[char]
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
145 if record:
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
146 if record[5]:
25219
8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113
Walter Dörwald <walter@livinglogic.de>
parents: 16091
diff changeset
147 decomp = record[5].split()
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
148 if len(decomp) > 19:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
149 raise Exception, "character %x has a decomposition too large for nfd_nfkd" % char
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
150 # prefix
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
151 if decomp[0][0] == "<":
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
152 prefix = decomp.pop(0)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
153 else:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
154 prefix = ""
14835
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
155 try:
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
156 i = decomp_prefix.index(prefix)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
157 except ValueError:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
158 i = len(decomp_prefix)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
159 decomp_prefix.append(prefix)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
160 prefix = i
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
161 assert prefix < 256
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
162 # content
59640
8aa5530e8407 Remove py3k deprecation warnings from these Unicode tools.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 56507
diff changeset
163 decomp = [prefix + (len(decomp)<<8)] + [int(s, 16) for s in decomp]
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
164 # Collect NFC pairs
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
165 if not prefix and len(decomp) == 3 and \
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
166 char not in unicode.exclusions and \
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
167 unicode.table[decomp[1]][3] == "0":
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
168 p, l, r = decomp
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
169 comp_first[l] = 1
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
170 comp_last[r] = 1
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
171 comp_pairs.append((l,r,char))
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
172 try:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
173 i = decomp_data.index(decomp)
14835
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
174 except ValueError:
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
175 i = len(decomp_data)
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
176 decomp_data.extend(decomp)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
177 decomp_size = decomp_size + len(decomp) * 2
14835
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
178 else:
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
179 i = 0
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
180 decomp_index[char] = i
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
181
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
182 f = l = 0
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
183 comp_first_ranges = []
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
184 comp_last_ranges = []
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
185 prev_f = prev_l = None
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
186 for i in unicode.chars:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
187 if comp_first[i] is not None:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
188 comp_first[i] = f
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
189 f += 1
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
190 if prev_f is None:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
191 prev_f = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
192 elif prev_f[1]+1 == i:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
193 prev_f = prev_f[0],i
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
194 else:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
195 comp_first_ranges.append(prev_f)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
196 prev_f = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
197 if comp_last[i] is not None:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
198 comp_last[i] = l
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
199 l += 1
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
200 if prev_l is None:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
201 prev_l = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
202 elif prev_l[1]+1 == i:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
203 prev_l = prev_l[0],i
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
204 else:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
205 comp_last_ranges.append(prev_l)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
206 prev_l = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
207 comp_first_ranges.append(prev_f)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
208 comp_last_ranges.append(prev_l)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
209 total_first = f
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
210 total_last = l
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
211
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
212 comp_data = [0]*(total_first*total_last)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
213 for f,l,char in comp_pairs:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
214 f = comp_first[f]
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
215 l = comp_last[l]
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
216 comp_data[f*total_last+l] = char
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
217
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
218 print len(table), "unique properties"
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
219 print len(decomp_prefix), "unique decomposition prefixes"
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
220 print len(decomp_data), "unique decomposition entries:",
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
221 print decomp_size, "bytes"
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
222 print total_first, "first characters in NFC"
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
223 print total_last, "last characters in NFC"
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
224 print len(comp_pairs), "NFC pairs"
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
225
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
226 print "--- Writing", FILE, "..."
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
227
15339
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
228 fp = open(FILE, "w")
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
229 print >>fp, "/* this file was generated by %s %s */" % (SCRIPT, VERSION)
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
230 print >>fp
26397
b27c1cad77ea Add unidata_version. Bump generator version number.
Martin v. Löwis <martin@v.loewis.de>
parents: 26394
diff changeset
231 print >>fp, '#define UNIDATA_VERSION "%s"' % UNIDATA_VERSION
15339
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
232 print >>fp, "/* a list of unique database records */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
233 print >>fp, \
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
234 "const _PyUnicode_DatabaseRecord _PyUnicode_Database_Records[] = {"
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
235 for item in table:
53553
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 53400
diff changeset
236 print >>fp, " {%d, %d, %d, %d, %d, %d}," % item
15339
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
237 print >>fp, "};"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
238 print >>fp
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
239
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
240 print >>fp, "/* Reindexing of NFC first characters. */"
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
241 print >>fp, "#define TOTAL_FIRST",total_first
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
242 print >>fp, "#define TOTAL_LAST",total_last
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
243 print >>fp, "struct reindex{int start;short count,index;};"
47678
a35da94e05fc Make more symbols static.
Martin v. Löwis <martin@v.loewis.de>
parents: 36940
diff changeset
244 print >>fp, "static struct reindex nfc_first[] = {"
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
245 for start,end in comp_first_ranges:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
246 print >>fp," { %d, %d, %d}," % (start,end-start,comp_first[start])
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
247 print >>fp," {0,0,0}"
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
248 print >>fp,"};\n"
47678
a35da94e05fc Make more symbols static.
Martin v. Löwis <martin@v.loewis.de>
parents: 36940
diff changeset
249 print >>fp, "static struct reindex nfc_last[] = {"
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
250 for start,end in comp_last_ranges:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
251 print >>fp," { %d, %d, %d}," % (start,end-start,comp_last[start])
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
252 print >>fp," {0,0,0}"
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
253 print >>fp,"};\n"
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
254
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
255 # FIXME: <fl> the following tables could be made static, and
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
256 # the support code moved into unicodedatabase.c
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
257
15339
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
258 print >>fp, "/* string literals */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
259 print >>fp, "const char *_PyUnicode_CategoryNames[] = {"
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
260 for name in CATEGORY_NAMES:
15339
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
261 print >>fp, " \"%s\"," % name
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
262 print >>fp, " NULL"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
263 print >>fp, "};"
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
264
15339
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
265 print >>fp, "const char *_PyUnicode_BidirectionalNames[] = {"
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
266 for name in BIDIRECTIONAL_NAMES:
15339
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
267 print >>fp, " \"%s\"," % name
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
268 print >>fp, " NULL"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
269 print >>fp, "};"
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
270
32951
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32197
diff changeset
271 print >>fp, "const char *_PyUnicode_EastAsianWidthNames[] = {"
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32197
diff changeset
272 for name in EASTASIANWIDTH_NAMES:
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32197
diff changeset
273 print >>fp, " \"%s\"," % name
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32197
diff changeset
274 print >>fp, " NULL"
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32197
diff changeset
275 print >>fp, "};"
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32197
diff changeset
276
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
277 print >>fp, "static const char *decomp_prefix[] = {"
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
278 for name in decomp_prefix:
15339
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
279 print >>fp, " \"%s\"," % name
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
280 print >>fp, " NULL"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
281 print >>fp, "};"
14835
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
282
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
283 # split record index table
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
284 index1, index2, shift = splitbins(index, trace)
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
285
15339
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
286 print >>fp, "/* index tables for the database records */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
287 print >>fp, "#define SHIFT", shift
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
288 Array("index1", index1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
289 Array("index2", index2).dump(fp, trace)
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
290
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
291 # split decomposition index table
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
292 index1, index2, shift = splitbins(decomp_index, trace)
14835
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14834
diff changeset
293
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
294 print >>fp, "/* decomposition data */"
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
295 Array("decomp_data", decomp_data).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
296
15339
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
297 print >>fp, "/* index tables for the decomposition data */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
298 print >>fp, "#define DECOMP_SHIFT", shift
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
299 Array("decomp_index1", index1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
300 Array("decomp_index2", index2).dump(fp, trace)
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
301
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
302 index, index2, shift = splitbins(comp_data, trace)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
303 print >>fp, "/* NFC pairs */"
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
304 print >>fp, "#define COMP_SHIFT", shift
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
305 Array("comp_index", index).dump(fp, trace)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
306 Array("comp_data", index2).dump(fp, trace)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
307
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
308 # Generate delta tables for old versions
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
309 for version, table, normalization in unicode.changed:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
310 cversion = version.replace(".","_")
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
311 records = [table[0]]
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
312 cache = {table[0]:0}
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
313 index = [0] * len(table)
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
314 for i, record in enumerate(table):
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
315 try:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
316 index[i] = cache[record]
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
317 except KeyError:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
318 index[i] = cache[record] = len(records)
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
319 records.append(record)
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
320 index1, index2, shift = splitbins(index, trace)
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
321 print >>fp, "static const change_record change_records_%s[] = {" % cversion
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
322 for record in records:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
323 print >>fp, "\t{ %s }," % ", ".join(map(str,record))
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
324 print >>fp, "};"
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
325 Array("changes_%s_index" % cversion, index1).dump(fp, trace)
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
326 Array("changes_%s_data" % cversion, index2).dump(fp, trace)
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
327 print >>fp, "static const change_record* get_change_%s(Py_UCS4 n)" % cversion
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
328 print >>fp, "{"
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
329 print >>fp, "\tint index;"
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
330 print >>fp, "\tif (n >= 0x110000) index = 0;"
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
331 print >>fp, "\telse {"
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
332 print >>fp, "\t\tindex = changes_%s_index[n>>%d];" % (cversion, shift)
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
333 print >>fp, "\t\tindex = changes_%s_data[(index<<%d)+(n & %d)];" % \
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
334 (cversion, shift, ((1<<shift)-1))
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
335 print >>fp, "\t}"
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
336 print >>fp, "\treturn change_records_%s+index;" % cversion
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
337 print >>fp, "}\n"
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
338 print >>fp, "static Py_UCS4 normalization_%s(Py_UCS4 n)" % cversion
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
339 print >>fp, "{"
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
340 print >>fp, "\tswitch(n) {"
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
341 for k, v in normalization:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
342 print >>fp, "\tcase %s: return 0x%s;" % (hex(k), v)
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
343 print >>fp, "\tdefault: return 0;"
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
344 print >>fp, "\t}\n}\n"
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
345
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
346 fp.close()
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
347
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
348 # --------------------------------------------------------------------
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
349 # unicode character type tables
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
350
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
351 def makeunicodetype(unicode, trace):
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
352
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
353 FILE = "Objects/unicodetype_db.h"
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
354
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
355 print "--- Preparing", FILE, "..."
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
356
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
357 # extract unicode types
14852
463faf01685e unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14849
diff changeset
358 dummy = (0, 0, 0, 0, 0, 0)
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
359 table = [dummy]
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
360 cache = {0: dummy}
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
361 index = [0] * len(unicode.chars)
56406
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
362 numeric = {}
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
363 spaces = []
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
364 linebreaks = []
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
365
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
366 for char in unicode.chars:
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
367 record = unicode.table[char]
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
368 if record:
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
369 # extract database properties
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
370 category = record[2]
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
371 bidirectional = record[4]
60074
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
372 properties = record[16]
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
373 flags = 0
49124
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 47678
diff changeset
374 delta = True
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
375 if category in ["Lm", "Lt", "Lu", "Ll", "Lo"]:
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
376 flags |= ALPHA_MASK
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
377 if category == "Ll":
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
378 flags |= LOWER_MASK
60074
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
379 if 'Line_Break' in properties or bidirectional == "B":
14852
463faf01685e unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14849
diff changeset
380 flags |= LINEBREAK_MASK
56406
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
381 linebreaks.append(char)
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
382 if category == "Zs" or bidirectional in ("WS", "B", "S"):
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
383 flags |= SPACE_MASK
56406
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
384 spaces.append(char)
14856
83aff1ea5b70 - don't set the titlecase flag for uppercase letters (sorry, tim)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14852
diff changeset
385 if category == "Lt":
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
386 flags |= TITLE_MASK
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
387 if category == "Lu":
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
388 flags |= UPPER_MASK
49124
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 47678
diff changeset
389 # use delta predictor for upper/lower/title if it fits
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
390 if record[12]:
53400
f08f776082ec Merged revisions 71894 via svnmerge from
Walter Dörwald <walter@livinglogic.de>
parents: 49124
diff changeset
391 upper = int(record[12], 16)
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
392 else:
53400
f08f776082ec Merged revisions 71894 via svnmerge from
Walter Dörwald <walter@livinglogic.de>
parents: 49124
diff changeset
393 upper = char
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
394 if record[13]:
53400
f08f776082ec Merged revisions 71894 via svnmerge from
Walter Dörwald <walter@livinglogic.de>
parents: 49124
diff changeset
395 lower = int(record[13], 16)
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
396 else:
53400
f08f776082ec Merged revisions 71894 via svnmerge from
Walter Dörwald <walter@livinglogic.de>
parents: 49124
diff changeset
397 lower = char
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
398 if record[14]:
53400
f08f776082ec Merged revisions 71894 via svnmerge from
Walter Dörwald <walter@livinglogic.de>
parents: 49124
diff changeset
399 title = int(record[14], 16)
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
400 else:
53400
f08f776082ec Merged revisions 71894 via svnmerge from
Walter Dörwald <walter@livinglogic.de>
parents: 49124
diff changeset
401 # UCD.html says that a missing title char means that
f08f776082ec Merged revisions 71894 via svnmerge from
Walter Dörwald <walter@livinglogic.de>
parents: 49124
diff changeset
402 # it defaults to the uppercase character, not to the
f08f776082ec Merged revisions 71894 via svnmerge from
Walter Dörwald <walter@livinglogic.de>
parents: 49124
diff changeset
403 # character itself. Apparently, in the current UCD (5.x)
f08f776082ec Merged revisions 71894 via svnmerge from
Walter Dörwald <walter@livinglogic.de>
parents: 49124
diff changeset
404 # this feature is never used
f08f776082ec Merged revisions 71894 via svnmerge from
Walter Dörwald <walter@livinglogic.de>
parents: 49124
diff changeset
405 title = upper
f08f776082ec Merged revisions 71894 via svnmerge from
Walter Dörwald <walter@livinglogic.de>
parents: 49124
diff changeset
406 upper_d = upper - char
f08f776082ec Merged revisions 71894 via svnmerge from
Walter Dörwald <walter@livinglogic.de>
parents: 49124
diff changeset
407 lower_d = lower - char
f08f776082ec Merged revisions 71894 via svnmerge from
Walter Dörwald <walter@livinglogic.de>
parents: 49124
diff changeset
408 title_d = title - char
f08f776082ec Merged revisions 71894 via svnmerge from
Walter Dörwald <walter@livinglogic.de>
parents: 49124
diff changeset
409 if -32768 <= upper_d <= 32767 and \
f08f776082ec Merged revisions 71894 via svnmerge from
Walter Dörwald <walter@livinglogic.de>
parents: 49124
diff changeset
410 -32768 <= lower_d <= 32767 and \
f08f776082ec Merged revisions 71894 via svnmerge from
Walter Dörwald <walter@livinglogic.de>
parents: 49124
diff changeset
411 -32768 <= title_d <= 32767:
f08f776082ec Merged revisions 71894 via svnmerge from
Walter Dörwald <walter@livinglogic.de>
parents: 49124
diff changeset
412 # use deltas
f08f776082ec Merged revisions 71894 via svnmerge from
Walter Dörwald <walter@livinglogic.de>
parents: 49124
diff changeset
413 upper = upper_d & 0xffff
f08f776082ec Merged revisions 71894 via svnmerge from
Walter Dörwald <walter@livinglogic.de>
parents: 49124
diff changeset
414 lower = lower_d & 0xffff
f08f776082ec Merged revisions 71894 via svnmerge from
Walter Dörwald <walter@livinglogic.de>
parents: 49124
diff changeset
415 title = title_d & 0xffff
f08f776082ec Merged revisions 71894 via svnmerge from
Walter Dörwald <walter@livinglogic.de>
parents: 49124
diff changeset
416 else:
49124
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 47678
diff changeset
417 flags |= NODELTA_MASK
14852
463faf01685e unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14849
diff changeset
418 # decimal digit, integer digit
463faf01685e unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14849
diff changeset
419 decimal = 0
463faf01685e unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14849
diff changeset
420 if record[6]:
463faf01685e unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14849
diff changeset
421 flags |= DECIMAL_MASK
463faf01685e unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14849
diff changeset
422 decimal = int(record[6])
463faf01685e unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14849
diff changeset
423 digit = 0
463faf01685e unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14849
diff changeset
424 if record[7]:
463faf01685e unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14849
diff changeset
425 flags |= DIGIT_MASK
463faf01685e unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14849
diff changeset
426 digit = int(record[7])
56406
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
427 if record[8]:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
428 flags |= NUMERIC_MASK
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
429 numeric.setdefault(record[8], []).append(char)
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
430 item = (
32197
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 26397
diff changeset
431 upper, lower, title, decimal, digit, flags
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
432 )
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
433 # add entry to index and item tables
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
434 i = cache.get(item)
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
435 if i is None:
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
436 cache[item] = i = len(table)
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
437 table.append(item)
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
438 index[char] = i
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
439
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
440 print len(table), "unique character type entries"
56406
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
441 print sum(map(len, numeric.values())), "numeric code points"
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
442 print len(spaces), "whitespace code points"
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
443 print len(linebreaks), "linebreak code points"
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
444
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
445 print "--- Writing", FILE, "..."
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
446
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
447 fp = open(FILE, "w")
15339
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
448 print >>fp, "/* this file was generated by %s %s */" % (SCRIPT, VERSION)
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
449 print >>fp
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
450 print >>fp, "/* a list of unique character type descriptors */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
451 print >>fp, "const _PyUnicode_TypeRecord _PyUnicode_TypeRecords[] = {"
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
452 for item in table:
15339
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
453 print >>fp, " {%d, %d, %d, %d, %d, %d}," % item
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
454 print >>fp, "};"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
455 print >>fp
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
456
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
457 # split decomposition index table
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
458 index1, index2, shift = splitbins(index, trace)
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
459
15339
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
460 print >>fp, "/* type indexes */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use
Fred Drake <fdrake@acm.org>
parents: 14856
diff changeset
461 print >>fp, "#define SHIFT", shift
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
462 Array("index1", index1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
463 Array("index2", index2).dump(fp, trace)
14849
92cdbc0f55d5 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
464
56406
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
465 # Generate code for _PyUnicode_ToNumeric()
59640
8aa5530e8407 Remove py3k deprecation warnings from these Unicode tools.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 56507
diff changeset
466 numeric_items = sorted(numeric.items())
56406
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
467 print >>fp, '/* Returns the numeric value as double for Unicode characters'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
468 print >>fp, ' * having this property, -1.0 otherwise.'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
469 print >>fp, ' */'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
470 print >>fp, 'double _PyUnicode_ToNumeric(Py_UNICODE ch)'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
471 print >>fp, '{'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
472 print >>fp, ' switch (ch) {'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
473 for value, codepoints in numeric_items:
56507
7fe8a8313a5b #7112: Fix compilation warning in unicodetype_db.h
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 56406
diff changeset
474 # Turn text into float literals
7fe8a8313a5b #7112: Fix compilation warning in unicodetype_db.h
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 56406
diff changeset
475 parts = value.split('/')
7fe8a8313a5b #7112: Fix compilation warning in unicodetype_db.h
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 56406
diff changeset
476 parts = [repr(float(part)) for part in parts]
7fe8a8313a5b #7112: Fix compilation warning in unicodetype_db.h
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 56406
diff changeset
477 value = '/'.join(parts)
7fe8a8313a5b #7112: Fix compilation warning in unicodetype_db.h
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 56406
diff changeset
478
56406
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
479 haswide = False
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
480 hasnonewide = False
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
481 codepoints.sort()
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
482 for codepoint in codepoints:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
483 if codepoint < 0x10000:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
484 hasnonewide = True
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
485 if codepoint >= 0x10000 and not haswide:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
486 print >>fp, '#ifdef Py_UNICODE_WIDE'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
487 haswide = True
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
488 print >>fp, ' case 0x%04X:' % (codepoint,)
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
489 if haswide and hasnonewide:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
490 print >>fp, '#endif'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
491 print >>fp, ' return (double) %s;' % (value,)
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
492 if haswide and not hasnonewide:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
493 print >>fp, '#endif'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
494 print >>fp,' }'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
495 print >>fp,' return -1.0;'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
496 print >>fp,'}'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
497 print >>fp
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
498
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
499 # Generate code for _PyUnicode_IsWhitespace()
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
500 print >>fp, "/* Returns 1 for Unicode characters having the bidirectional"
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
501 print >>fp, " * type 'WS', 'B' or 'S' or the category 'Zs', 0 otherwise."
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
502 print >>fp, " */"
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
503 print >>fp, 'int _PyUnicode_IsWhitespace(register const Py_UNICODE ch)'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
504 print >>fp, '{'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
505 print >>fp, '#ifdef WANT_WCTYPE_FUNCTIONS'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
506 print >>fp, ' return iswspace(ch);'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
507 print >>fp, '#else'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
508 print >>fp, ' switch (ch) {'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
509
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
510 haswide = False
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
511 hasnonewide = False
59640
8aa5530e8407 Remove py3k deprecation warnings from these Unicode tools.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 56507
diff changeset
512 for codepoint in sorted(spaces):
56406
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
513 if codepoint < 0x10000:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
514 hasnonewide = True
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
515 if codepoint >= 0x10000 and not haswide:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
516 print >>fp, '#ifdef Py_UNICODE_WIDE'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
517 haswide = True
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
518 print >>fp, ' case 0x%04X:' % (codepoint,)
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
519 if haswide and hasnonewide:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
520 print >>fp, '#endif'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
521 print >>fp, ' return 1;'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
522 if haswide and not hasnonewide:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
523 print >>fp, '#endif'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
524
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
525 print >>fp,' }'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
526 print >>fp,' return 0;'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
527 print >>fp, '#endif'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
528 print >>fp,'}'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
529 print >>fp
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
530
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
531 # Generate code for _PyUnicode_IsLinebreak()
60074
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
532 print >>fp, "/* Returns 1 for Unicode characters having the line break"
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
533 print >>fp, " * property 'BK', 'CR', 'LF' or 'NL' or having bidirectional"
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
534 print >>fp, " * type 'B', 0 otherwise."
56406
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
535 print >>fp, " */"
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
536 print >>fp, 'int _PyUnicode_IsLinebreak(register const Py_UNICODE ch)'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
537 print >>fp, '{'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
538 print >>fp, ' switch (ch) {'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
539 haswide = False
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
540 hasnonewide = False
59640
8aa5530e8407 Remove py3k deprecation warnings from these Unicode tools.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 56507
diff changeset
541 for codepoint in sorted(linebreaks):
56406
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
542 if codepoint < 0x10000:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
543 hasnonewide = True
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
544 if codepoint >= 0x10000 and not haswide:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
545 print >>fp, '#ifdef Py_UNICODE_WIDE'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
546 haswide = True
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
547 print >>fp, ' case 0x%04X:' % (codepoint,)
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
548 if haswide and hasnonewide:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
549 print >>fp, '#endif'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
550 print >>fp, ' return 1;'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
551 if haswide and not hasnonewide:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
552 print >>fp, '#endif'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
553
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
554 print >>fp,' }'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
555 print >>fp,' return 0;'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
556 print >>fp,'}'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
557 print >>fp
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
558
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
559 fp.close()
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
560
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
561 # --------------------------------------------------------------------
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
562 # unicode name database
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
563
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
564 def makeunicodename(unicode, trace):
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
565
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
566 FILE = "Modules/unicodename_db.h"
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
567
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
568 print "--- Preparing", FILE, "..."
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
569
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
570 # collect names
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
571 names = [None] * len(unicode.chars)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
572
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
573 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
574 record = unicode.table[char]
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
575 if record:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
576 name = record[1].strip()
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
577 if name and name[0] != "<":
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
578 names[char] = name + chr(0)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
579
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
580 print len(filter(lambda n: n is not None, names)), "distinct names"
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
581
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
582 # collect unique words from names (note that we differ between
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
583 # words inside a sentence, and words ending a sentence. the
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
584 # latter includes the trailing null byte.
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
585
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
586 words = {}
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
587 n = b = 0
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
588 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
589 name = names[char]
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
590 if name:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
591 w = name.split()
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
592 b = b + len(name)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
593 n = n + len(w)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
594 for w in w:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
595 l = words.get(w)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
596 if l:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
597 l.append(None)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
598 else:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
599 words[w] = [len(words)]
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
600
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
601 print n, "words in text;", b, "bytes"
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
602
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
603 wordlist = words.items()
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
604
26394
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
605 # sort on falling frequency, then by name
59640
8aa5530e8407 Remove py3k deprecation warnings from these Unicode tools.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 56507
diff changeset
606 def word_key(a):
8aa5530e8407 Remove py3k deprecation warnings from these Unicode tools.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 56507
diff changeset
607 aword, alist = a
8aa5530e8407 Remove py3k deprecation warnings from these Unicode tools.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 56507
diff changeset
608 return -len(alist), aword
8aa5530e8407 Remove py3k deprecation warnings from these Unicode tools.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 56507
diff changeset
609 wordlist.sort(key=word_key)
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
610
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
611 # figure out how many phrasebook escapes we need
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
612 escapes = 0
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
613 while escapes * 256 < len(wordlist):
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
614 escapes = escapes + 1
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
615 print escapes, "escapes"
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
616
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
617 short = 256 - escapes
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
618
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
619 assert short > 0
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
620
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
621 print short, "short indexes in lexicon"
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
622
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
623 # statistics
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
624 n = 0
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
625 for i in range(short):
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
626 n = n + len(wordlist[i][1])
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
627 print n, "short indexes in phrasebook"
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
628
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
629 # pick the most commonly used words, and sort the rest on falling
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
630 # length (to maximize overlap)
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
631
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
632 wordlist, wordtail = wordlist[:short], wordlist[short:]
59640
8aa5530e8407 Remove py3k deprecation warnings from these Unicode tools.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 56507
diff changeset
633 wordtail.sort(key=lambda a: a[0], reverse=True)
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
634 wordlist.extend(wordtail)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
635
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
636 # generate lexicon from words
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
637
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
638 lexicon_offset = [0]
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
639 lexicon = ""
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
640 words = {}
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
641
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
642 # build a lexicon string
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
643 offset = 0
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
644 for w, x in wordlist:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
645 # encoding: bit 7 indicates last character in word (chr(128)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
646 # indicates the last character in an entire string)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
647 ww = w[:-1] + chr(ord(w[-1])+128)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
648 # reuse string tails, when possible
25219
8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113
Walter Dörwald <walter@livinglogic.de>
parents: 16091
diff changeset
649 o = lexicon.find(ww)
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
650 if o < 0:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
651 o = offset
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
652 lexicon = lexicon + ww
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
653 offset = offset + len(w)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
654 words[w] = len(lexicon_offset)
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
655 lexicon_offset.append(o)
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
656
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
657 lexicon = map(ord, lexicon)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
658
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
659 # generate phrasebook from names and lexicon
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
660 phrasebook = [0]
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
661 phrasebook_offset = [0] * len(unicode.chars)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
662 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
663 name = names[char]
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
664 if name:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
665 w = name.split()
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
666 phrasebook_offset[char] = len(phrasebook)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
667 for w in w:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
668 i = words[w]
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
669 if i < short:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
670 phrasebook.append(i)
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
671 else:
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
672 # store as two bytes
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
673 phrasebook.append((i>>8) + short)
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
674 phrasebook.append(i&255)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
675
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
676 assert getsize(phrasebook) == 1
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
677
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
678 #
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
679 # unicode name hash table
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
680
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
681 # extract names
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
682 data = []
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
683 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
684 record = unicode.table[char]
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
685 if record:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
686 name = record[1].strip()
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
687 if name and name[0] != "<":
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
688 data.append((name, char))
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
689
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
690 # the magic number 47 was chosen to minimize the number of
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
691 # collisions on the current data set. if you like, change it
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
692 # and see what happens...
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
693
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
694 codehash = Hash("code", data, 47)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
695
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
696 print "--- Writing", FILE, "..."
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
697
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
698 fp = open(FILE, "w")
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
699 print >>fp, "/* this file was generated by %s %s */" % (SCRIPT, VERSION)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
700 print >>fp
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
701 print >>fp, "#define NAME_MAXLEN", 256
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
702 print >>fp
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
703 print >>fp, "/* lexicon */"
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
704 Array("lexicon", lexicon).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
705 Array("lexicon_offset", lexicon_offset).dump(fp, trace)
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
706
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
707 # split decomposition index table
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
708 offset1, offset2, shift = splitbins(phrasebook_offset, trace)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
709
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
710 print >>fp, "/* code->name phrasebook */"
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
711 print >>fp, "#define phrasebook_shift", shift
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
712 print >>fp, "#define phrasebook_short", short
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
713
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
714 Array("phrasebook", phrasebook).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
715 Array("phrasebook_offset1", offset1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
716 Array("phrasebook_offset2", offset2).dump(fp, trace)
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
717
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
718 print >>fp, "/* name->code dictionary */"
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
719 codehash.dump(fp, trace)
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
720
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
721 fp.close()
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
722
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
723
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
724 def merge_old_version(version, new, old):
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
725 # Changes to exclusion file not implemented yet
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
726 if old.exclusions != new.exclusions:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
727 raise NotImplementedError, "exclusions differ"
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
728
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
729 # In these change records, 0xFF means "no change"
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
730 bidir_changes = [0xFF]*0x110000
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
731 category_changes = [0xFF]*0x110000
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
732 decimal_changes = [0xFF]*0x110000
49124
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 47678
diff changeset
733 mirrored_changes = [0xFF]*0x110000
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
734 # In numeric data, 0 means "no change",
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
735 # -1 means "did not have a numeric value
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
736 numeric_changes = [0] * 0x110000
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
737 # normalization_changes is a list of key-value pairs
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
738 normalization_changes = []
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
739 for i in range(0x110000):
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
740 if new.table[i] is None:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
741 # Characters unassigned in the new version ought to
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
742 # be unassigned in the old one
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
743 assert old.table[i] is None
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
744 continue
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
745 # check characters unassigned in the old version
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
746 if old.table[i] is None:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
747 # category 0 is "unassigned"
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
748 category_changes[i] = 0
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
749 continue
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
750 # check characters that differ
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
751 if old.table[i] != new.table[i]:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
752 for k in range(len(old.table[i])):
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
753 if old.table[i][k] != new.table[i][k]:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
754 value = old.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
755 if k == 2:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
756 #print "CATEGORY",hex(i), old.table[i][k], new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
757 category_changes[i] = CATEGORY_NAMES.index(value)
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
758 elif k == 4:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
759 #print "BIDIR",hex(i), old.table[i][k], new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
760 bidir_changes[i] = BIDIRECTIONAL_NAMES.index(value)
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
761 elif k == 5:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
762 #print "DECOMP",hex(i), old.table[i][k], new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
763 # We assume that all normalization changes are in 1:1 mappings
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
764 assert " " not in value
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
765 normalization_changes.append((i, value))
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
766 elif k == 6:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
767 #print "DECIMAL",hex(i), old.table[i][k], new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
768 # we only support changes where the old value is a single digit
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
769 assert value in "0123456789"
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
770 decimal_changes[i] = int(value)
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
771 elif k == 8:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
772 # print "NUMERIC",hex(i), `old.table[i][k]`, new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
773 # Since 0 encodes "no change", the old value is better not 0
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
774 if not value:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
775 numeric_changes[i] = -1
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
776 else:
56406
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
777 numeric_changes[i] = float(value)
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
778 assert numeric_changes[i] not in (0, -1)
49124
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 47678
diff changeset
779 elif k == 9:
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 47678
diff changeset
780 if value == 'Y':
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 47678
diff changeset
781 mirrored_changes[i] = '1'
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 47678
diff changeset
782 else:
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 47678
diff changeset
783 mirrored_changes[i] = '0'
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
784 elif k == 11:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
785 # change to ISO comment, ignore
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
786 pass
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
787 elif k == 12:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
788 # change to simple uppercase mapping; ignore
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
789 pass
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
790 elif k == 13:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
791 # change to simple lowercase mapping; ignore
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
792 pass
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
793 elif k == 14:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
794 # change to simple titlecase mapping; ignore
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
795 pass
60074
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
796 elif k == 16:
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
797 # change to properties; not yet
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
798 pass
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
799 else:
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
800 class Difference(Exception):pass
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
801 raise Difference, (hex(i), k, old.table[i], new.table[i])
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
802 new.changed.append((version, zip(bidir_changes, category_changes,
49124
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 47678
diff changeset
803 decimal_changes, mirrored_changes,
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 47678
diff changeset
804 numeric_changes),
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
805 normalization_changes))
36937
df43c04d2554 Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 36924
diff changeset
806
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
807
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
808 # --------------------------------------------------------------------
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
809 # the following support code is taken from the unidb utilities
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
810 # Copyright (c) 1999-2000 by Secret Labs AB
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
811
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
812 # load a unicode-data file from disk
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
813
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
814 class UnicodeData:
60074
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
815 # Record structure:
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
816 # [ID, name, category, combining, bidi, decomp, (6)
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
817 # decimal, digit, numeric, bidi-mirrored, Unicode-1-name, (11)
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
818 # ISO-comment, uppercase, lowercase, titlecase, ea-width, (16)
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
819 # properties] (17)
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
820
56406
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
821 def __init__(self, filename, exclusions, eastasianwidth, unihan,
60074
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
822 derivednormalizationprops=None, linebreakprops=None,
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
823 expand=1):
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
824 self.changed = []
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
825 file = open(filename)
25852
1f5f6481886d Update to Unicode 3.2 database.
Martin v. Löwis <martin@v.loewis.de>
parents: 25219
diff changeset
826 table = [None] * 0x110000
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
827 while 1:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
828 s = file.readline()
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
829 if not s:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
830 break
25219
8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113
Walter Dörwald <walter@livinglogic.de>
parents: 16091
diff changeset
831 s = s.strip().split(";")
8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113
Walter Dörwald <walter@livinglogic.de>
parents: 16091
diff changeset
832 char = int(s[0], 16)
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
833 table[char] = s
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
834
26394
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
835 # expand first-last ranges
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
836 if expand:
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
837 field = None
26394
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
838 for i in range(0, 0x110000):
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
839 s = table[i]
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
840 if s:
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
841 if s[1][-6:] == "First>":
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
842 s[1] = ""
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
843 field = s
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
844 elif s[1][-5:] == "Last>":
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
845 s[1] = ""
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
846 field = None
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
847 elif field:
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
848 f2 = field[:]
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
849 f2[0] = "%X" % i
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 32951
diff changeset
850 table[i] = f2
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
851
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
852 # public attributes
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
853 self.filename = filename
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
854 self.table = table
25852
1f5f6481886d Update to Unicode 3.2 database.
Martin v. Löwis <martin@v.loewis.de>
parents: 25219
diff changeset
855 self.chars = range(0x110000) # unicode 3.2
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
856
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
857 file = open(exclusions)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
858 self.exclusions = {}
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
859 for s in file:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
860 s = s.strip()
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
861 if not s:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
862 continue
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
863 if s[0] == '#':
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
864 continue
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
865 char = int(s.split()[0],16)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
866 self.exclusions[char] = 1
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 25860
diff changeset
867
32197
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 26397
diff changeset
868 widths = [None] * 0x110000
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 26397
diff changeset
869 for s in open(eastasianwidth):
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 26397
diff changeset
870 s = s.strip()
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 26397
diff changeset
871 if not s:
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 26397
diff changeset
872 continue
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 26397
diff changeset
873 if s[0] == '#':
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 26397
diff changeset
874 continue
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 26397
diff changeset
875 s = s.split()[0].split(';')
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 26397
diff changeset
876 if '..' in s[0]:
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 26397
diff changeset
877 first, last = [int(c, 16) for c in s[0].split('..')]
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 26397
diff changeset
878 chars = range(first, last+1)
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 26397
diff changeset
879 else:
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 26397
diff changeset
880 chars = [int(s[0], 16)]
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 26397
diff changeset
881 for char in chars:
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 26397
diff changeset
882 widths[char] = s[1]
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 26397
diff changeset
883 for i in range(0, 0x110000):
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 26397
diff changeset
884 if table[i] is not None:
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 26397
diff changeset
885 table[i].append(widths[i])
60074
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
886
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
887 for i in range(0, 0x110000):
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
888 if table[i] is not None:
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
889 table[i].append(set())
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
890 if linebreakprops:
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
891 for s in open(linebreakprops):
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
892 s = s.partition('#')[0]
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
893 s = [i.strip() for i in s.split(';')]
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
894 if len(s) < 2 or s[1] not in MANDATORY_LINE_BREAKS:
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
895 continue
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
896 if '..' not in s[0]:
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
897 first = last = int(s[0], 16)
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
898 else:
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
899 first, last = [int(c, 16) for c in s[0].split('..')]
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
900 for char in range(first, last+1):
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
901 table[char][-1].add('Line_Break')
66e757d8ea93 #7643: Unicode codepoints VT (0x0B) and FF (0x0C) are linebreaks according to Unicode Standard Annex #14.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59693
diff changeset
902
53553
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 53400
diff changeset
903 if derivednormalizationprops:
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 53400
diff changeset
904 quickchecks = [0] * 0x110000 # default is Yes
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 53400
diff changeset
905 qc_order = 'NFD_QC NFKD_QC NFC_QC NFKC_QC'.split()
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 53400
diff changeset
906 for s in open(derivednormalizationprops):
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 53400
diff changeset
907 if '#' in s:
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 53400
diff changeset
908 s = s[:s.index('#')]
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 53400
diff changeset
909 s = [i.strip() for i in s.split(';')]
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 53400
diff changeset
910 if len(s) < 2 or s[1] not in qc_order:
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 53400
diff changeset
911 continue
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 53400
diff changeset
912 quickcheck = 'MN'.index(s[2]) + 1 # Maybe or No
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 53400
diff changeset
913 quickcheck_shift = qc_order.index(s[1])*2
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 53400
diff changeset
914 quickcheck <<= quickcheck_shift
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 53400
diff changeset
915 if '..' not in s[0]:
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 53400
diff changeset
916 first = last = int(s[0], 16)
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 53400
diff changeset
917 else:
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 53400
diff changeset
918 first, last = [int(c, 16) for c in s[0].split('..')]
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 53400
diff changeset
919 for char in range(first, last+1):
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 53400
diff changeset
920 assert not (quickchecks[char]>>quickcheck_shift)&3
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 53400
diff changeset
921 quickchecks[char] |= quickcheck
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 53400
diff changeset
922 for i in range(0, 0x110000):
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 53400
diff changeset
923 if table[i] is not None:
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 53400
diff changeset
924 table[i].append(quickchecks[i])
32197
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 26397
diff changeset
925
56406
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
926 for line in open(unihan):
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
927 if not line.startswith('U+'):
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
928 continue
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
929 code, tag, value = line.split(None, 3)[:3]
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
930 if tag not in ('kAccountingNumeric', 'kPrimaryNumeric',
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
931 'kOtherNumeric'):
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
932 continue
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
933 value = value.strip().replace(',', '')
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
934 i = int(code[2:], 16)
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
935 # Patch the numeric field
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
936 if table[i] is not None:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
937 table[i][8] = value
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
938
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
939 def uselatin1(self):
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
940 # restrict character range to ISO Latin 1
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
941 self.chars = range(256)
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
942
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
943 # hash table tools
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
944
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
945 # this is a straight-forward reimplementation of Python's built-in
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
946 # dictionary type, using a static data structure, and a custom string
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
947 # hash algorithm.
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
948
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
949 def myhash(s, magic):
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
950 h = 0
25219
8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113
Walter Dörwald <walter@livinglogic.de>
parents: 16091
diff changeset
951 for c in map(ord, s.upper()):
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
952 h = (h * magic) + c
26394
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
953 ix = h & 0xff000000L
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
954 if ix:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
955 h = (h ^ ((ix>>24) & 0xff)) & 0x00ffffff
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
956 return h
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
957
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
958 SIZES = [
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
959 (4,3), (8,3), (16,3), (32,5), (64,3), (128,3), (256,29), (512,17),
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
960 (1024,9), (2048,5), (4096,83), (8192,27), (16384,43), (32768,3),
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
961 (65536,45), (131072,9), (262144,39), (524288,39), (1048576,9),
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
962 (2097152,5), (4194304,3), (8388608,33), (16777216,27)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
963 ]
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
964
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
965 class Hash:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
966 def __init__(self, name, data, magic):
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
967 # turn a (key, value) list into a static hash table structure
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
968
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
969 # determine table size
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
970 for size, poly in SIZES:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
971 if size > len(data):
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
972 poly = size + poly
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
973 break
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
974 else:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
975 raise AssertionError, "ran out of polynominals"
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
976
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
977 print size, "slots in hash table"
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
978
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
979 table = [None] * size
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
980
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
981 mask = size-1
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
982
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
983 n = 0
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
984
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
985 hash = myhash
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
986
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
987 # initialize hash table
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
988 for key, value in data:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
989 h = hash(key, magic)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
990 i = (~h) & mask
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
991 v = table[i]
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
992 if v is None:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
993 table[i] = value
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
994 continue
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
995 incr = (h ^ (h >> 3)) & mask;
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
996 if not incr:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
997 incr = mask
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
998 while 1:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
999 n = n + 1
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1000 i = (i + incr) & mask
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1001 v = table[i]
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1002 if v is None:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1003 table[i] = value
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1004 break
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1005 incr = incr << 1
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1006 if incr > mask:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1007 incr = incr ^ poly
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1008
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1009 print n, "collisions"
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1010 self.collisions = n
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1011
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1012 for i in range(len(table)):
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1013 if table[i] is None:
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1014 table[i] = 0
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1015
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1016 self.data = Array(name + "_hash", table)
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1017 self.magic = magic
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1018 self.name = name
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1019 self.size = size
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1020 self.poly = poly
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1021
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
1022 def dump(self, file, trace):
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1023 # write data to file, as a C array
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
1024 self.data.dump(file, trace)
16080
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1025 file.write("#define %s_magic %d\n" % (self.name, self.magic))
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1026 file.write("#define %s_size %d\n" % (self.name, self.size))
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1027 file.write("#define %s_poly %d\n" % (self.name, self.poly))
93f836432c2b forgot to check in the new makeunicodedata.py script
Fredrik Lundh <fredrik@pythonware.com>
parents: 15376
diff changeset
1028
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1029 # stuff to deal with arrays of unsigned integers
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1030
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1031 class Array:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1032
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1033 def __init__(self, name, data):
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1034 self.name = name
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1035 self.data = data
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1036
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
1037 def dump(self, file, trace=0):
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1038 # write data to file, as a C array
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1039 size = getsize(self.data)
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
1040 if trace:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 16080
diff changeset
1041 print >>sys.stderr, self.name+":", size*len(self.data), "bytes"
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1042 file.write("static ")
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1043 if size == 1:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1044 file.write("unsigned char")
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1045 elif size == 2:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1046 file.write("unsigned short")
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1047 else:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1048 file.write("unsigned int")
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1049 file.write(" " + self.name + "[] = {\n")
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1050 if self.data:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1051 s = " "
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1052 for item in self.data:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1053 i = str(item) + ", "
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1054 if len(s) + len(i) > 78:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1055 file.write(s + "\n")
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1056 s = " " + i
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1057 else:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1058 s = s + i
25219
8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113
Walter Dörwald <walter@livinglogic.de>
parents: 16091
diff changeset
1059 if s.strip():
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1060 file.write(s + "\n")
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1061 file.write("};\n\n")
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1062
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1063 def getsize(data):
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1064 # return smallest possible integer size for the given array
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1065 maxdata = max(data)
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1066 if maxdata < 256:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1067 return 1
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1068 elif maxdata < 65536:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1069 return 2
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1070 else:
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1071 return 4
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1072
14834
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1073 def splitbins(t, trace=0):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1074 """t, trace=0 -> (t1, t2, shift). Split a table to save space.
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1075
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1076 t is a sequence of ints. This function can be useful to save space if
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1077 many of the ints are the same. t1 and t2 are lists of ints, and shift
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1078 is an int, chosen to minimize the combined size of t1 and t2 (in C
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1079 code), and where for each i in range(len(t)),
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1080 t[i] == t2[(t1[i >> shift] << shift) + (i & mask)]
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1081 where mask is a bitmask isolating the last "shift" bits.
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1082
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
1083 If optional arg trace is non-zero (default zero), progress info
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
1084 is printed to sys.stderr. The higher the value, the more info
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
1085 you'll get.
14834
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1086 """
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1087
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1088 if trace:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1089 def dump(t1, t2, shift, bytes):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1090 print >>sys.stderr, "%d+%d bins at shift %d; %d bytes" % (
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1091 len(t1), len(t2), shift, bytes)
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1092 print >>sys.stderr, "Size of original table:", len(t)*getsize(t), \
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1093 "bytes"
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1094 n = len(t)-1 # last valid index
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1095 maxshift = 0 # the most we can shift n and still have something left
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1096 if n > 0:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1097 while n >> 1:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1098 n >>= 1
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1099 maxshift += 1
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1100 del n
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1101 bytes = sys.maxint # smallest total size so far
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1102 t = tuple(t) # so slices can be dict keys
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1103 for shift in range(maxshift + 1):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1104 t1 = []
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1105 t2 = []
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1106 size = 2**shift
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1107 bincache = {}
14834
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1108 for i in range(0, len(t), size):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1109 bin = t[i:i+size]
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1110 index = bincache.get(bin)
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1111 if index is None:
14834
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1112 index = len(t2)
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1113 bincache[bin] = index
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1114 t2.extend(bin)
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1115 t1.append(index >> shift)
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1116 # determine memory size
14834
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1117 b = len(t1)*getsize(t1) + len(t2)*getsize(t2)
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
1118 if trace > 1:
14834
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1119 dump(t1, t2, shift, b)
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1120 if b < bytes:
14834
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1121 best = t1, t2, shift
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1122 bytes = b
14834
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1123 t1, t2, shift = best
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1124 if trace:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1125 print >>sys.stderr, "Best:",
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1126 dump(t1, t2, shift, bytes)
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1127 if __debug__:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1128 # exhaustively verify that the decomposition is correct
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1129 mask = ~((~0) << shift) # i.e., low-bit mask of shift bits
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1130 for i in xrange(len(t)):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1131 assert t[i] == t2[(t1[i >> shift] << shift) + (i & mask)]
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it
Tim Peters <tim.peters@gmail.com>
parents: 14825
diff changeset
1132 return best
14825
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1133
9fd030af2fe0 unicode database compression, step 1:
Fredrik Lundh <fredrik@pythonware.com>
parents:
diff changeset
1134 if __name__ == "__main__":
15376
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last
Fredrik Lundh <fredrik@pythonware.com>
parents: 15339
diff changeset
1135 maketables(1)