cpython-lru_cache-weakref: Tools/unicode/makeunicodedata.py annotate

annotate Tools/unicode/makeunicodedata.py @ 27998:e27f58a50737 legacy-trunk

SF bug #690377: Dud footnote reference in distutils docs

author	Raymond Hettinger <python@rcn.com>
date	Fri, 21 Feb 2003 02:47:09 +0000
parents	b27c1cad77ea
children	3a84e23b042f

rev	line source
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1 #
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	2 # (re)generate unicode property and type databases
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	3 #
26397 b27c1cad77ea Add unidata_version. Bump generator version number. Martin v. Löwis <martin@v.loewis.de> parents: 26394 diff changeset	4 # this script converts a unicode 3.2 database file to
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	5 # Modules/unicodedata_db.h, Modules/unicodename_db.h,
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	6 # and Objects/unicodetype_db.h
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	7 #
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	8 # history:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	9 # 2000-09-24 fl created (based on bits and pieces from unidb)
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	10 # 2000-09-25 fl merged tim's splitbin fixes, separate decomposition table
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	11 # 2000-09-25 fl added character type table
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	12 # 2000-09-26 fl added LINEBREAK, DECIMAL, and DIGIT flags/fields (2.0)
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	13 # 2000-11-03 fl expand first/last ranges
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	14 # 2001-01-19 fl added character name tables (2.1)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	15 # 2001-01-21 fl added decomp compression; dynamic phrasebook threshold
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	16 # 2002-09-11 wd use string methods
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	17 # 2002-10-18 mvl update to Unicode 3.2
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	18 # 2002-10-22 mvl generate NFC tables
26394 491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	19 # 2002-11-24 mvl expand all ranges, sort names version-independently
26397 b27c1cad77ea Add unidata_version. Bump generator version number. Martin v. Löwis <martin@v.loewis.de> parents: 26394 diff changeset	20 # 2002-11-25 mvl add UNIDATA_VERSION
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	21 #
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	22 # written by Fredrik Lundh (fredrik@pythonware.com)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	23 #
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	24
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	25 import sys
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	26
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	27 SCRIPT = sys.argv[0]
26397 b27c1cad77ea Add unidata_version. Bump generator version number. Martin v. Löwis <martin@v.loewis.de> parents: 26394 diff changeset	28 VERSION = "2.2"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	29
26397 b27c1cad77ea Add unidata_version. Bump generator version number. Martin v. Löwis <martin@v.loewis.de> parents: 26394 diff changeset	30 # The Unicode Database
b27c1cad77ea Add unidata_version. Bump generator version number. Martin v. Löwis <martin@v.loewis.de> parents: 26394 diff changeset	31 UNIDATA_VERSION = "3.2.0"
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	32 UNICODE_DATA = "UnicodeData.txt"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	33 COMPOSITION_EXCLUSIONS = "CompositionExclusions.txt"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	34
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	35 CATEGORY_NAMES = [ "Cn", "Lu", "Ll", "Lt", "Mn", "Mc", "Me", "Nd",
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	36 "Nl", "No", "Zs", "Zl", "Zp", "Cc", "Cf", "Cs", "Co", "Cn", "Lm",
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	37 "Lo", "Pc", "Pd", "Ps", "Pe", "Pi", "Pf", "Po", "Sm", "Sc", "Sk",
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	38 "So" ]
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	39
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	40 BIDIRECTIONAL_NAMES = [ "", "L", "LRE", "LRO", "R", "AL", "RLE", "RLO",
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	41 "PDF", "EN", "ES", "ET", "AN", "CS", "NSM", "BN", "B", "S", "WS",
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	42 "ON" ]
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	43
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	44 # note: should match definitions in Objects/unicodectype.c
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	45 ALPHA_MASK = 0x01
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	46 DECIMAL_MASK = 0x02
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	47 DIGIT_MASK = 0x04
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	48 LOWER_MASK = 0x08
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	49 LINEBREAK_MASK = 0x10
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	50 SPACE_MASK = 0x20
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	51 TITLE_MASK = 0x40
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	52 UPPER_MASK = 0x80
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	53
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	54 def maketables(trace=0):
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	55
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	56 print "--- Reading", UNICODE_DATA, "..."
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	57
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	58 unicode = UnicodeData(UNICODE_DATA, COMPOSITION_EXCLUSIONS)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	59
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	60 print len(filter(None, unicode.table)), "characters"
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	61
16091 45c065668ea5 Unicode nits: Don't include unicodedatabase.h no more. And make sure Fredrik Lundh <fredrik@pythonware.com> parents: 16090 diff changeset	62 makeunicodename(unicode, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	63 makeunicodedata(unicode, trace)
16091 45c065668ea5 Unicode nits: Don't include unicodedatabase.h no more. And make sure Fredrik Lundh <fredrik@pythonware.com> parents: 16090 diff changeset	64 makeunicodetype(unicode, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	65
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	66 # --------------------------------------------------------------------
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	67 # unicode character properties
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	68
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	69 def makeunicodedata(unicode, trace):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	70
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	71 dummy = (0, 0, 0, 0)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	72 table = [dummy]
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	73 cache = {0: dummy}
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	74 index = [0] * len(unicode.chars)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	75
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	76 FILE = "Modules/unicodedata_db.h"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	77
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	78 print "--- Preparing", FILE, "..."
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	79
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	80 # 1) database properties
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	81
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	82 for char in unicode.chars:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	83 record = unicode.table[char]
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	84 if record:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	85 # extract database properties
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	86 category = CATEGORY_NAMES.index(record[2])
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	87 combining = int(record[3])
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	88 bidirectional = BIDIRECTIONAL_NAMES.index(record[4])
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	89 mirrored = record[9] == "Y"
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	90 item = (
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	91 category, combining, bidirectional, mirrored
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	92 )
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	93 # add entry to index and item tables
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	94 i = cache.get(item)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	95 if i is None:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	96 cache[item] = i = len(table)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	97 table.append(item)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	98 index[char] = i
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	99
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	100 # 2) decomposition data
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	101
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	102 decomp_data = [0]
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	103 decomp_prefix = [""]
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	104 decomp_index = [0] * len(unicode.chars)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	105 decomp_size = 0
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	106
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	107 comp_pairs = []
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	108 comp_first = [None] * len(unicode.chars)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	109 comp_last = [None] * len(unicode.chars)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	110
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	111 for char in unicode.chars:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	112 record = unicode.table[char]
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	113 if record:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	114 if record[5]:
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	115 decomp = record[5].split()
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	116 # prefix
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	117 if decomp[0][0] == "<":
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	118 prefix = decomp.pop(0)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	119 else:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	120 prefix = ""
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	121 try:
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	122 i = decomp_prefix.index(prefix)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	123 except ValueError:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	124 i = len(decomp_prefix)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	125 decomp_prefix.append(prefix)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	126 prefix = i
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	127 assert prefix < 256
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	128 # content
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	129 decomp = [prefix + (len(decomp)<<8)] +\
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	130 map(lambda s: int(s, 16), decomp)
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	131 # Collect NFC pairs
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	132 if not prefix and len(decomp) == 3 and \
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	133 char not in unicode.exclusions and \
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	134 unicode.table[decomp[1]][3] == "0":
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	135 p, l, r = decomp
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	136 comp_first[l] = 1
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	137 comp_last[r] = 1
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	138 comp_pairs.append((l,r,char))
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	139 try:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	140 i = decomp_data.index(decomp)
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	141 except ValueError:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	142 i = len(decomp_data)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	143 decomp_data.extend(decomp)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	144 decomp_size = decomp_size + len(decomp) * 2
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	145 else:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	146 i = 0
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	147 decomp_index[char] = i
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	148
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	149 f = l = 0
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	150 comp_first_ranges = []
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	151 comp_last_ranges = []
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	152 prev_f = prev_l = None
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	153 for i in unicode.chars:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	154 if comp_first[i] is not None:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	155 comp_first[i] = f
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	156 f += 1
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	157 if prev_f is None:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	158 prev_f = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	159 elif prev_f[1]+1 == i:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	160 prev_f = prev_f[0],i
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	161 else:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	162 comp_first_ranges.append(prev_f)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	163 prev_f = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	164 if comp_last[i] is not None:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	165 comp_last[i] = l
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	166 l += 1
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	167 if prev_l is None:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	168 prev_l = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	169 elif prev_l[1]+1 == i:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	170 prev_l = prev_l[0],i
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	171 else:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	172 comp_last_ranges.append(prev_l)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	173 prev_l = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	174 comp_first_ranges.append(prev_f)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	175 comp_last_ranges.append(prev_l)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	176 total_first = f
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	177 total_last = l
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	178
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	179 comp_data = [0](total_firsttotal_last)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	180 for f,l,char in comp_pairs:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	181 f = comp_first[f]
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	182 l = comp_last[l]
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	183 comp_data[f*total_last+l] = char
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	184
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	185 print len(table), "unique properties"
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	186 print len(decomp_prefix), "unique decomposition prefixes"
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	187 print len(decomp_data), "unique decomposition entries:",
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	188 print decomp_size, "bytes"
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	189 print total_first, "first characters in NFC"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	190 print total_last, "last characters in NFC"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	191 print len(comp_pairs), "NFC pairs"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	192
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	193 print "--- Writing", FILE, "..."
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	194
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	195 fp = open(FILE, "w")
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	196 print >>fp, "/* this file was generated by %s %s */" % (SCRIPT, VERSION)
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	197 print >>fp
26397 b27c1cad77ea Add unidata_version. Bump generator version number. Martin v. Löwis <martin@v.loewis.de> parents: 26394 diff changeset	198 print >>fp, '#define UNIDATA_VERSION "%s"' % UNIDATA_VERSION
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	199 print >>fp, "/* a list of unique database records */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	200 print >>fp, \
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	201 "const _PyUnicode_DatabaseRecord _PyUnicode_Database_Records[] = {"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	202 for item in table:
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	203 print >>fp, " {%d, %d, %d, %d}," % item
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	204 print >>fp, "};"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	205 print >>fp
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	206
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	207 print >>fp, "/* Reindexing of NFC first characters. */"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	208 print >>fp, "#define TOTAL_FIRST",total_first
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	209 print >>fp, "#define TOTAL_LAST",total_last
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	210 print >>fp, "struct reindex{int start;short count,index;};"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	211 print >>fp, "struct reindex nfc_first[] = {"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	212 for start,end in comp_first_ranges:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	213 print >>fp," { %d, %d, %d}," % (start,end-start,comp_first[start])
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	214 print >>fp," {0,0,0}"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	215 print >>fp,"};\n"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	216 print >>fp, "struct reindex nfc_last[] = {"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	217 for start,end in comp_last_ranges:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	218 print >>fp," { %d, %d, %d}," % (start,end-start,comp_last[start])
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	219 print >>fp," {0,0,0}"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	220 print >>fp,"};\n"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	221
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	222 # FIXME: <fl> the following tables could be made static, and
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	223 # the support code moved into unicodedatabase.c
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	224
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	225 print >>fp, "/* string literals */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	226 print >>fp, "const char *_PyUnicode_CategoryNames[] = {"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	227 for name in CATEGORY_NAMES:
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	228 print >>fp, " \"%s\"," % name
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	229 print >>fp, " NULL"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	230 print >>fp, "};"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	231
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	232 print >>fp, "const char *_PyUnicode_BidirectionalNames[] = {"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	233 for name in BIDIRECTIONAL_NAMES:
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	234 print >>fp, " \"%s\"," % name
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	235 print >>fp, " NULL"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	236 print >>fp, "};"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	237
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	238 print >>fp, "static const char *decomp_prefix[] = {"
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	239 for name in decomp_prefix:
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	240 print >>fp, " \"%s\"," % name
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	241 print >>fp, " NULL"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	242 print >>fp, "};"
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	243
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	244 # split record index table
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	245 index1, index2, shift = splitbins(index, trace)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	246
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	247 print >>fp, "/* index tables for the database records */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	248 print >>fp, "#define SHIFT", shift
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	249 Array("index1", index1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	250 Array("index2", index2).dump(fp, trace)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	251
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	252 # split decomposition index table
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	253 index1, index2, shift = splitbins(decomp_index, trace)
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	254
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	255 print >>fp, "/* decomposition data */"
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	256 Array("decomp_data", decomp_data).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	257
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	258 print >>fp, "/* index tables for the decomposition data */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	259 print >>fp, "#define DECOMP_SHIFT", shift
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	260 Array("decomp_index1", index1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	261 Array("decomp_index2", index2).dump(fp, trace)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	262
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	263 index, index2, shift = splitbins(comp_data, trace)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	264 print >>fp, "/* NFC pairs */"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	265 print >>fp, "#define COMP_SHIFT", shift
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	266 Array("comp_index", index).dump(fp, trace)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	267 Array("comp_data", index2).dump(fp, trace)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	268
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	269 fp.close()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	270
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	271 # --------------------------------------------------------------------
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	272 # unicode character type tables
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	273
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	274 def makeunicodetype(unicode, trace):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	275
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	276 FILE = "Objects/unicodetype_db.h"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	277
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	278 print "--- Preparing", FILE, "..."
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	279
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	280 # extract unicode types
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	281 dummy = (0, 0, 0, 0, 0, 0)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	282 table = [dummy]
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	283 cache = {0: dummy}
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	284 index = [0] * len(unicode.chars)
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	285
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	286 for char in unicode.chars:
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	287 record = unicode.table[char]
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	288 if record:
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	289 # extract database properties
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	290 category = record[2]
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	291 bidirectional = record[4]
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	292 flags = 0
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	293 if category in ["Lm", "Lt", "Lu", "Ll", "Lo"]:
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	294 flags \|= ALPHA_MASK
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	295 if category == "Ll":
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	296 flags \|= LOWER_MASK
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	297 if category == "Zl" or bidirectional == "B":
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	298 flags \|= LINEBREAK_MASK
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	299 if category == "Zs" or bidirectional in ("WS", "B", "S"):
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	300 flags \|= SPACE_MASK
14856 83aff1ea5b70 - don't set the titlecase flag for uppercase letters (sorry, tim) Fredrik Lundh <fredrik@pythonware.com> parents: 14852 diff changeset	301 if category == "Lt":
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	302 flags \|= TITLE_MASK
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	303 if category == "Lu":
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	304 flags \|= UPPER_MASK
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	305 # use delta predictor for upper/lower/title
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	306 if record[12]:
25860 823e947e2450 Verify that lower-higher case delta are 16-bit. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	307 upper = int(record[12], 16) - char
823e947e2450 Verify that lower-higher case delta are 16-bit. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	308 assert -32768 <= upper <= 32767
823e947e2450 Verify that lower-higher case delta are 16-bit. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	309 upper = upper & 0xffff
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	310 else:
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	311 upper = 0
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	312 if record[13]:
25860 823e947e2450 Verify that lower-higher case delta are 16-bit. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	313 lower = int(record[13], 16) - char
823e947e2450 Verify that lower-higher case delta are 16-bit. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	314 assert -32768 <= lower <= 32767
823e947e2450 Verify that lower-higher case delta are 16-bit. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	315 lower = lower & 0xffff
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	316 else:
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	317 lower = 0
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	318 if record[14]:
25860 823e947e2450 Verify that lower-higher case delta are 16-bit. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	319 title = int(record[14], 16) - char
823e947e2450 Verify that lower-higher case delta are 16-bit. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	320 assert -32768 <= lower <= 32767
823e947e2450 Verify that lower-higher case delta are 16-bit. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	321 title = title & 0xffff
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	322 else:
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	323 title = 0
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	324 # decimal digit, integer digit
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	325 decimal = 0
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	326 if record[6]:
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	327 flags \|= DECIMAL_MASK
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	328 decimal = int(record[6])
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	329 digit = 0
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	330 if record[7]:
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	331 flags \|= DIGIT_MASK
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	332 digit = int(record[7])
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	333 item = (
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	334 flags, upper, lower, title, decimal, digit
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	335 )
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	336 # add entry to index and item tables
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	337 i = cache.get(item)
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	338 if i is None:
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	339 cache[item] = i = len(table)
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	340 table.append(item)
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	341 index[char] = i
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	342
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	343 print len(table), "unique character type entries"
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	344
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	345 print "--- Writing", FILE, "..."
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	346
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	347 fp = open(FILE, "w")
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	348 print >>fp, "/* this file was generated by %s %s */" % (SCRIPT, VERSION)
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	349 print >>fp
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	350 print >>fp, "/* a list of unique character type descriptors */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	351 print >>fp, "const _PyUnicode_TypeRecord _PyUnicode_TypeRecords[] = {"
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	352 for item in table:
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	353 print >>fp, " {%d, %d, %d, %d, %d, %d}," % item
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	354 print >>fp, "};"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	355 print >>fp
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	356
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	357 # split decomposition index table
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	358 index1, index2, shift = splitbins(index, trace)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	359
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	360 print >>fp, "/* type indexes */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	361 print >>fp, "#define SHIFT", shift
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	362 Array("index1", index1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	363 Array("index2", index2).dump(fp, trace)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	364
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	365 fp.close()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	366
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	367 # --------------------------------------------------------------------
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	368 # unicode name database
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	369
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	370 def makeunicodename(unicode, trace):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	371
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	372 FILE = "Modules/unicodename_db.h"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	373
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	374 print "--- Preparing", FILE, "..."
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	375
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	376 # collect names
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	377 names = [None] * len(unicode.chars)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	378
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	379 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	380 record = unicode.table[char]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	381 if record:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	382 name = record[1].strip()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	383 if name and name[0] != "<":
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	384 names[char] = name + chr(0)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	385
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	386 print len(filter(lambda n: n is not None, names)), "distinct names"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	387
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	388 # collect unique words from names (note that we differ between
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	389 # words inside a sentence, and words ending a sentence. the
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	390 # latter includes the trailing null byte.
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	391
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	392 words = {}
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	393 n = b = 0
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	394 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	395 name = names[char]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	396 if name:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	397 w = name.split()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	398 b = b + len(name)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	399 n = n + len(w)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	400 for w in w:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	401 l = words.get(w)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	402 if l:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	403 l.append(None)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	404 else:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	405 words[w] = [len(words)]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	406
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	407 print n, "words in text;", b, "bytes"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	408
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	409 wordlist = words.items()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	410
26394 491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	411 # sort on falling frequency, then by name
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	412 def cmpwords((aword, alist),(bword, blist)):
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	413 r = -cmp(len(alist),len(blist))
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	414 if r:
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	415 return r
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	416 return cmp(aword, bword)
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	417 wordlist.sort(cmpwords)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	418
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	419 # figure out how many phrasebook escapes we need
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	420 escapes = 0
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	421 while escapes * 256 < len(wordlist):
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	422 escapes = escapes + 1
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	423 print escapes, "escapes"
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	424
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	425 short = 256 - escapes
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	426
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	427 assert short > 0
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	428
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	429 print short, "short indexes in lexicon"
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	430
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	431 # statistics
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	432 n = 0
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	433 for i in range(short):
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	434 n = n + len(wordlist[i][1])
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	435 print n, "short indexes in phrasebook"
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	436
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	437 # pick the most commonly used words, and sort the rest on falling
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	438 # length (to maximize overlap)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	439
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	440 wordlist, wordtail = wordlist[:short], wordlist[short:]
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	441 wordtail.sort(lambda a, b: len(b[0])-len(a[0]))
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	442 wordlist.extend(wordtail)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	443
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	444 # generate lexicon from words
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	445
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	446 lexicon_offset = [0]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	447 lexicon = ""
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	448 words = {}
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	449
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	450 # build a lexicon string
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	451 offset = 0
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	452 for w, x in wordlist:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	453 # encoding: bit 7 indicates last character in word (chr(128)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	454 # indicates the last character in an entire string)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	455 ww = w[:-1] + chr(ord(w[-1])+128)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	456 # reuse string tails, when possible
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	457 o = lexicon.find(ww)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	458 if o < 0:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	459 o = offset
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	460 lexicon = lexicon + ww
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	461 offset = offset + len(w)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	462 words[w] = len(lexicon_offset)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	463 lexicon_offset.append(o)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	464
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	465 lexicon = map(ord, lexicon)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	466
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	467 # generate phrasebook from names and lexicon
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	468 phrasebook = [0]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	469 phrasebook_offset = [0] * len(unicode.chars)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	470 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	471 name = names[char]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	472 if name:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	473 w = name.split()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	474 phrasebook_offset[char] = len(phrasebook)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	475 for w in w:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	476 i = words[w]
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	477 if i < short:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	478 phrasebook.append(i)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	479 else:
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	480 # store as two bytes
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	481 phrasebook.append((i>>8) + short)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	482 phrasebook.append(i&255)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	483
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	484 assert getsize(phrasebook) == 1
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	485
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	486 #
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	487 # unicode name hash table
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	488
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	489 # extract names
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	490 data = []
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	491 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	492 record = unicode.table[char]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	493 if record:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	494 name = record[1].strip()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	495 if name and name[0] != "<":
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	496 data.append((name, char))
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	497
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	498 # the magic number 47 was chosen to minimize the number of
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	499 # collisions on the current data set. if you like, change it
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	500 # and see what happens...
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	501
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	502 codehash = Hash("code", data, 47)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	503
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	504 print "--- Writing", FILE, "..."
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	505
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	506 fp = open(FILE, "w")
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	507 print >>fp, "/* this file was generated by %s %s */" % (SCRIPT, VERSION)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	508 print >>fp
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	509 print >>fp, "#define NAME_MAXLEN", 256
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	510 print >>fp
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	511 print >>fp, "/* lexicon */"
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	512 Array("lexicon", lexicon).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	513 Array("lexicon_offset", lexicon_offset).dump(fp, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	514
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	515 # split decomposition index table
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	516 offset1, offset2, shift = splitbins(phrasebook_offset, trace)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	517
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	518 print >>fp, "/* code->name phrasebook */"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	519 print >>fp, "#define phrasebook_shift", shift
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	520 print >>fp, "#define phrasebook_short", short
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	521
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	522 Array("phrasebook", phrasebook).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	523 Array("phrasebook_offset1", offset1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	524 Array("phrasebook_offset2", offset2).dump(fp, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	525
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	526 print >>fp, "/* name->code dictionary */"
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	527 codehash.dump(fp, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	528
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	529 fp.close()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	530
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	531 # --------------------------------------------------------------------
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	532 # the following support code is taken from the unidb utilities
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	533 # Copyright (c) 1999-2000 by Secret Labs AB
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	534
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	535 # load a unicode-data file from disk
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	536
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	537 import sys
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	538
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	539 class UnicodeData:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	540
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	541 def __init__(self, filename, exclusions, expand=1):
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	542 file = open(filename)
25852 1f5f6481886d Update to Unicode 3.2 database. Martin v. Löwis <martin@v.loewis.de> parents: 25219 diff changeset	543 table = [None] * 0x110000
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	544 while 1:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	545 s = file.readline()
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	546 if not s:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	547 break
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	548 s = s.strip().split(";")
8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	549 char = int(s[0], 16)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	550 table[char] = s
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	551
26394 491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	552 # expand first-last ranges
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	553 if expand:
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	554 field = None
26394 491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	555 for i in range(0, 0x110000):
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	556 s = table[i]
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	557 if s:
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	558 if s[1][-6:] == "First>":
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	559 s[1] = ""
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	560 field = s[:]
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	561 elif s[1][-5:] == "Last>":
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	562 s[1] = ""
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	563 field = None
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	564 elif field:
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	565 field[0] = hex(i)
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	566 table[i] = field
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	567
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	568 # public attributes
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	569 self.filename = filename
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	570 self.table = table
25852 1f5f6481886d Update to Unicode 3.2 database. Martin v. Löwis <martin@v.loewis.de> parents: 25219 diff changeset	571 self.chars = range(0x110000) # unicode 3.2
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	572
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	573 file = open(exclusions)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	574 self.exclusions = {}
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	575 for s in file:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	576 s = s.strip()
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	577 if not s:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	578 continue
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	579 if s[0] == '#':
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	580 continue
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	581 char = int(s.split()[0],16)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	582 self.exclusions[char] = 1
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	583
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	584 def uselatin1(self):
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	585 # restrict character range to ISO Latin 1
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	586 self.chars = range(256)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	587
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	588 # hash table tools
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	589
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	590 # this is a straight-forward reimplementation of Python's built-in
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	591 # dictionary type, using a static data structure, and a custom string
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	592 # hash algorithm.
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	593
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	594 def myhash(s, magic):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	595 h = 0
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	596 for c in map(ord, s.upper()):
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	597 h = (h * magic) + c
26394 491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	598 ix = h & 0xff000000L
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	599 if ix:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	600 h = (h ^ ((ix>>24) & 0xff)) & 0x00ffffff
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	601 return h
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	602
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	603 SIZES = [
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	604 (4,3), (8,3), (16,3), (32,5), (64,3), (128,3), (256,29), (512,17),
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	605 (1024,9), (2048,5), (4096,83), (8192,27), (16384,43), (32768,3),
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	606 (65536,45), (131072,9), (262144,39), (524288,39), (1048576,9),
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	607 (2097152,5), (4194304,3), (8388608,33), (16777216,27)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	608 ]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	609
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	610 class Hash:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	611 def __init__(self, name, data, magic):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	612 # turn a (key, value) list into a static hash table structure
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	613
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	614 # determine table size
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	615 for size, poly in SIZES:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	616 if size > len(data):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	617 poly = size + poly
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	618 break
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	619 else:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	620 raise AssertionError, "ran out of polynominals"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	621
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	622 print size, "slots in hash table"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	623
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	624 table = [None] * size
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	625
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	626 mask = size-1
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	627
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	628 n = 0
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	629
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	630 hash = myhash
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	631
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	632 # initialize hash table
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	633 for key, value in data:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	634 h = hash(key, magic)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	635 i = (~h) & mask
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	636 v = table[i]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	637 if v is None:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	638 table[i] = value
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	639 continue
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	640 incr = (h ^ (h >> 3)) & mask;
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	641 if not incr:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	642 incr = mask
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	643 while 1:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	644 n = n + 1
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	645 i = (i + incr) & mask
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	646 v = table[i]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	647 if v is None:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	648 table[i] = value
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	649 break
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	650 incr = incr << 1
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	651 if incr > mask:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	652 incr = incr ^ poly
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	653
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	654 print n, "collisions"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	655 self.collisions = n
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	656
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	657 for i in range(len(table)):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	658 if table[i] is None:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	659 table[i] = 0
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	660
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	661 self.data = Array(name + "_hash", table)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	662 self.magic = magic
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	663 self.name = name
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	664 self.size = size
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	665 self.poly = poly
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	666
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	667 def dump(self, file, trace):
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	668 # write data to file, as a C array
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	669 self.data.dump(file, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	670 file.write("#define %s_magic %d\n" % (self.name, self.magic))
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	671 file.write("#define %s_size %d\n" % (self.name, self.size))
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	672 file.write("#define %s_poly %d\n" % (self.name, self.poly))
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	673
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	674 # stuff to deal with arrays of unsigned integers
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	675
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	676 class Array:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	677
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	678 def __init__(self, name, data):
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	679 self.name = name
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	680 self.data = data
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	681
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	682 def dump(self, file, trace=0):
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	683 # write data to file, as a C array
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	684 size = getsize(self.data)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	685 if trace:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	686 print >>sys.stderr, self.name+":", size*len(self.data), "bytes"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	687 file.write("static ")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	688 if size == 1:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	689 file.write("unsigned char")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	690 elif size == 2:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	691 file.write("unsigned short")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	692 else:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	693 file.write("unsigned int")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	694 file.write(" " + self.name + "[] = {\n")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	695 if self.data:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	696 s = " "
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	697 for item in self.data:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	698 i = str(item) + ", "
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	699 if len(s) + len(i) > 78:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	700 file.write(s + "\n")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	701 s = " " + i
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	702 else:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	703 s = s + i
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	704 if s.strip():
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	705 file.write(s + "\n")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	706 file.write("};\n\n")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	707
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	708 def getsize(data):
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	709 # return smallest possible integer size for the given array
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	710 maxdata = max(data)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	711 if maxdata < 256:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	712 return 1
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	713 elif maxdata < 65536:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	714 return 2
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	715 else:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	716 return 4
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	717
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	718 def splitbins(t, trace=0):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	719 """t, trace=0 -> (t1, t2, shift). Split a table to save space.
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	720
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	721 t is a sequence of ints. This function can be useful to save space if
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	722 many of the ints are the same. t1 and t2 are lists of ints, and shift
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	723 is an int, chosen to minimize the combined size of t1 and t2 (in C
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	724 code), and where for each i in range(len(t)),
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	725 t[i] == t2[(t1[i >> shift] << shift) + (i & mask)]
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	726 where mask is a bitmask isolating the last "shift" bits.
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	727
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	728 If optional arg trace is non-zero (default zero), progress info
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	729 is printed to sys.stderr. The higher the value, the more info
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	730 you'll get.
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	731 """
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	732
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	733 import sys
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	734 if trace:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	735 def dump(t1, t2, shift, bytes):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	736 print >>sys.stderr, "%d+%d bins at shift %d; %d bytes" % (
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	737 len(t1), len(t2), shift, bytes)
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	738 print >>sys.stderr, "Size of original table:", len(t)*getsize(t), \
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	739 "bytes"
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	740 n = len(t)-1 # last valid index
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	741 maxshift = 0 # the most we can shift n and still have something left
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	742 if n > 0:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	743 while n >> 1:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	744 n >>= 1
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	745 maxshift += 1
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	746 del n
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	747 bytes = sys.maxint # smallest total size so far
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	748 t = tuple(t) # so slices can be dict keys
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	749 for shift in range(maxshift + 1):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	750 t1 = []
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	751 t2 = []
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	752 size = 2**shift
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	753 bincache = {}
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	754 for i in range(0, len(t), size):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	755 bin = t[i:i+size]
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	756 index = bincache.get(bin)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	757 if index is None:
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	758 index = len(t2)
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	759 bincache[bin] = index
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	760 t2.extend(bin)
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	761 t1.append(index >> shift)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	762 # determine memory size
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	763 b = len(t1)getsize(t1) + len(t2)getsize(t2)
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	764 if trace > 1:
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	765 dump(t1, t2, shift, b)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	766 if b < bytes:
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	767 best = t1, t2, shift
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	768 bytes = b
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	769 t1, t2, shift = best
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	770 if trace:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	771 print >>sys.stderr, "Best:",
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	772 dump(t1, t2, shift, bytes)
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	773 if __debug__:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	774 # exhaustively verify that the decomposition is correct
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	775 mask = ~((~0) << shift) # i.e., low-bit mask of shift bits
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	776 for i in xrange(len(t)):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	777 assert t[i] == t2[(t1[i >> shift] << shift) + (i & mask)]
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	778 return best
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	779
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	780 if __name__ == "__main__":
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	781 maketables(1)

Mercurial > cpython-lru_cache-weakref

annotate Tools/unicode/makeunicodedata.py @ 27998:e27f58a50737 legacy-trunk