cpython-lru_cache-weakref: Tools/unicode/makeunicodedata.py annotate

annotate Tools/unicode/makeunicodedata.py @ 58925:eecf1bef052d 2.6

Merged revisions 78207 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r78207 | ezio.melotti | 2010-02-17 01:26:09 +0200 (Wed, 17 Feb 2010) | 1 line #7930: fix stripid ........

author	Ezio Melotti <ezio.melotti@gmail.com>
date	Tue, 16 Feb 2010 23:29:44 +0000
parents	f08f776082ec
children	019244f4b0d0

rev	line source
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1 #
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	2 # (re)generate unicode property and type databases
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	3 #
26397 b27c1cad77ea Add unidata_version. Bump generator version number. Martin v. Löwis <martin@v.loewis.de> parents: 26394 diff changeset	4 # this script converts a unicode 3.2 database file to
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	5 # Modules/unicodedata_db.h, Modules/unicodename_db.h,
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	6 # and Objects/unicodetype_db.h
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	7 #
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	8 # history:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	9 # 2000-09-24 fl created (based on bits and pieces from unidb)
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	10 # 2000-09-25 fl merged tim's splitbin fixes, separate decomposition table
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	11 # 2000-09-25 fl added character type table
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	12 # 2000-09-26 fl added LINEBREAK, DECIMAL, and DIGIT flags/fields (2.0)
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	13 # 2000-11-03 fl expand first/last ranges
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	14 # 2001-01-19 fl added character name tables (2.1)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	15 # 2001-01-21 fl added decomp compression; dynamic phrasebook threshold
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	16 # 2002-09-11 wd use string methods
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	17 # 2002-10-18 mvl update to Unicode 3.2
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	18 # 2002-10-22 mvl generate NFC tables
26394 491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	19 # 2002-11-24 mvl expand all ranges, sort names version-independently
26397 b27c1cad77ea Add unidata_version. Bump generator version number. Martin v. Löwis <martin@v.loewis.de> parents: 26394 diff changeset	20 # 2002-11-25 mvl add UNIDATA_VERSION
32197 3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	21 # 2004-05-29 perky add east asian width information
36940 e4465d2737c4 Add changelog entry. Martin v. Löwis <martin@v.loewis.de> parents: 36937 diff changeset	22 # 2006-03-10 mvl update to Unicode 4.1; add UCD 3.2 delta
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	23 #
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	24 # written by Fredrik Lundh (fredrik@pythonware.com)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	25 #
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	26
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	27 import sys
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	28
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	29 SCRIPT = sys.argv[0]
49124 c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	30 VERSION = "2.6"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	31
26397 b27c1cad77ea Add unidata_version. Bump generator version number. Martin v. Löwis <martin@v.loewis.de> parents: 26394 diff changeset	32 # The Unicode Database
49124 c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	33 UNIDATA_VERSION = "5.1.0"
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	34 UNICODE_DATA = "UnicodeData%s.txt"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	35 COMPOSITION_EXCLUSIONS = "CompositionExclusions%s.txt"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	36 EASTASIAN_WIDTH = "EastAsianWidth%s.txt"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	37
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	38 old_versions = ["3.2.0"]
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	39
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	40 CATEGORY_NAMES = [ "Cn", "Lu", "Ll", "Lt", "Mn", "Mc", "Me", "Nd",
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	41 "Nl", "No", "Zs", "Zl", "Zp", "Cc", "Cf", "Cs", "Co", "Cn", "Lm",
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	42 "Lo", "Pc", "Pd", "Ps", "Pe", "Pi", "Pf", "Po", "Sm", "Sc", "Sk",
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	43 "So" ]
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	44
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	45 BIDIRECTIONAL_NAMES = [ "", "L", "LRE", "LRO", "R", "AL", "RLE", "RLO",
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	46 "PDF", "EN", "ES", "ET", "AN", "CS", "NSM", "BN", "B", "S", "WS",
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	47 "ON" ]
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	48
32951 024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	49 EASTASIANWIDTH_NAMES = [ "F", "H", "W", "Na", "A", "N" ]
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	50
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	51 # note: should match definitions in Objects/unicodectype.c
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	52 ALPHA_MASK = 0x01
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	53 DECIMAL_MASK = 0x02
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	54 DIGIT_MASK = 0x04
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	55 LOWER_MASK = 0x08
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	56 LINEBREAK_MASK = 0x10
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	57 SPACE_MASK = 0x20
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	58 TITLE_MASK = 0x40
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	59 UPPER_MASK = 0x80
49124 c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	60 NODELTA_MASK = 0x100
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	61
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	62 def maketables(trace=0):
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	63
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	64 print "--- Reading", UNICODE_DATA % "", "..."
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	65
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	66 version = ""
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	67 unicode = UnicodeData(UNICODE_DATA % version,
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	68 COMPOSITION_EXCLUSIONS % version,
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	69 EASTASIAN_WIDTH % version)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	70
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	71 print len(filter(None, unicode.table)), "characters"
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	72
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	73 for version in old_versions:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	74 print "--- Reading", UNICODE_DATA % ("-"+version), "..."
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	75 old_unicode = UnicodeData(UNICODE_DATA % ("-"+version),
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	76 COMPOSITION_EXCLUSIONS % ("-"+version),
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	77 EASTASIAN_WIDTH % ("-"+version))
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	78 print len(filter(None, old_unicode.table)), "characters"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	79 merge_old_version(version, unicode, old_unicode)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	80
16091 45c065668ea5 Unicode nits: Don't include unicodedatabase.h no more. And make sure Fredrik Lundh <fredrik@pythonware.com> parents: 16090 diff changeset	81 makeunicodename(unicode, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	82 makeunicodedata(unicode, trace)
16091 45c065668ea5 Unicode nits: Don't include unicodedatabase.h no more. And make sure Fredrik Lundh <fredrik@pythonware.com> parents: 16090 diff changeset	83 makeunicodetype(unicode, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	84
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	85 # --------------------------------------------------------------------
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	86 # unicode character properties
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	87
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	88 def makeunicodedata(unicode, trace):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	89
32951 024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	90 dummy = (0, 0, 0, 0, 0)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	91 table = [dummy]
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	92 cache = {0: dummy}
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	93 index = [0] * len(unicode.chars)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	94
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	95 FILE = "Modules/unicodedata_db.h"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	96
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	97 print "--- Preparing", FILE, "..."
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	98
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	99 # 1) database properties
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	100
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	101 for char in unicode.chars:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	102 record = unicode.table[char]
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	103 if record:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	104 # extract database properties
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	105 category = CATEGORY_NAMES.index(record[2])
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	106 combining = int(record[3])
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	107 bidirectional = BIDIRECTIONAL_NAMES.index(record[4])
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	108 mirrored = record[9] == "Y"
32951 024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	109 eastasianwidth = EASTASIANWIDTH_NAMES.index(record[15])
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	110 item = (
32951 024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	111 category, combining, bidirectional, mirrored, eastasianwidth
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	112 )
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	113 # add entry to index and item tables
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	114 i = cache.get(item)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	115 if i is None:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	116 cache[item] = i = len(table)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	117 table.append(item)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	118 index[char] = i
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	119
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	120 # 2) decomposition data
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	121
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	122 decomp_data = [0]
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	123 decomp_prefix = [""]
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	124 decomp_index = [0] * len(unicode.chars)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	125 decomp_size = 0
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	126
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	127 comp_pairs = []
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	128 comp_first = [None] * len(unicode.chars)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	129 comp_last = [None] * len(unicode.chars)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	130
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	131 for char in unicode.chars:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	132 record = unicode.table[char]
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	133 if record:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	134 if record[5]:
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	135 decomp = record[5].split()
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	136 if len(decomp) > 19:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	137 raise Exception, "character %x has a decomposition too large for nfd_nfkd" % char
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	138 # prefix
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	139 if decomp[0][0] == "<":
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	140 prefix = decomp.pop(0)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	141 else:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	142 prefix = ""
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	143 try:
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	144 i = decomp_prefix.index(prefix)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	145 except ValueError:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	146 i = len(decomp_prefix)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	147 decomp_prefix.append(prefix)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	148 prefix = i
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	149 assert prefix < 256
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	150 # content
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	151 decomp = [prefix + (len(decomp)<<8)] +\
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	152 map(lambda s: int(s, 16), decomp)
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	153 # Collect NFC pairs
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	154 if not prefix and len(decomp) == 3 and \
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	155 char not in unicode.exclusions and \
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	156 unicode.table[decomp[1]][3] == "0":
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	157 p, l, r = decomp
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	158 comp_first[l] = 1
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	159 comp_last[r] = 1
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	160 comp_pairs.append((l,r,char))
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	161 try:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	162 i = decomp_data.index(decomp)
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	163 except ValueError:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	164 i = len(decomp_data)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	165 decomp_data.extend(decomp)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	166 decomp_size = decomp_size + len(decomp) * 2
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	167 else:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	168 i = 0
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	169 decomp_index[char] = i
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	170
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	171 f = l = 0
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	172 comp_first_ranges = []
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	173 comp_last_ranges = []
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	174 prev_f = prev_l = None
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	175 for i in unicode.chars:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	176 if comp_first[i] is not None:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	177 comp_first[i] = f
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	178 f += 1
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	179 if prev_f is None:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	180 prev_f = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	181 elif prev_f[1]+1 == i:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	182 prev_f = prev_f[0],i
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	183 else:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	184 comp_first_ranges.append(prev_f)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	185 prev_f = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	186 if comp_last[i] is not None:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	187 comp_last[i] = l
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	188 l += 1
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	189 if prev_l is None:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	190 prev_l = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	191 elif prev_l[1]+1 == i:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	192 prev_l = prev_l[0],i
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	193 else:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	194 comp_last_ranges.append(prev_l)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	195 prev_l = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	196 comp_first_ranges.append(prev_f)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	197 comp_last_ranges.append(prev_l)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	198 total_first = f
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	199 total_last = l
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	200
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	201 comp_data = [0](total_firsttotal_last)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	202 for f,l,char in comp_pairs:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	203 f = comp_first[f]
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	204 l = comp_last[l]
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	205 comp_data[f*total_last+l] = char
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	206
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	207 print len(table), "unique properties"
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	208 print len(decomp_prefix), "unique decomposition prefixes"
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	209 print len(decomp_data), "unique decomposition entries:",
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	210 print decomp_size, "bytes"
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	211 print total_first, "first characters in NFC"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	212 print total_last, "last characters in NFC"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	213 print len(comp_pairs), "NFC pairs"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	214
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	215 print "--- Writing", FILE, "..."
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	216
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	217 fp = open(FILE, "w")
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	218 print >>fp, "/* this file was generated by %s %s */" % (SCRIPT, VERSION)
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	219 print >>fp
26397 b27c1cad77ea Add unidata_version. Bump generator version number. Martin v. Löwis <martin@v.loewis.de> parents: 26394 diff changeset	220 print >>fp, '#define UNIDATA_VERSION "%s"' % UNIDATA_VERSION
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	221 print >>fp, "/* a list of unique database records */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	222 print >>fp, \
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	223 "const _PyUnicode_DatabaseRecord _PyUnicode_Database_Records[] = {"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	224 for item in table:
32951 024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	225 print >>fp, " {%d, %d, %d, %d, %d}," % item
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	226 print >>fp, "};"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	227 print >>fp
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	228
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	229 print >>fp, "/* Reindexing of NFC first characters. */"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	230 print >>fp, "#define TOTAL_FIRST",total_first
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	231 print >>fp, "#define TOTAL_LAST",total_last
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	232 print >>fp, "struct reindex{int start;short count,index;};"
47678 a35da94e05fc Make more symbols static. Martin v. Löwis <martin@v.loewis.de> parents: 36940 diff changeset	233 print >>fp, "static struct reindex nfc_first[] = {"
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	234 for start,end in comp_first_ranges:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	235 print >>fp," { %d, %d, %d}," % (start,end-start,comp_first[start])
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	236 print >>fp," {0,0,0}"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	237 print >>fp,"};\n"
47678 a35da94e05fc Make more symbols static. Martin v. Löwis <martin@v.loewis.de> parents: 36940 diff changeset	238 print >>fp, "static struct reindex nfc_last[] = {"
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	239 for start,end in comp_last_ranges:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	240 print >>fp," { %d, %d, %d}," % (start,end-start,comp_last[start])
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	241 print >>fp," {0,0,0}"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	242 print >>fp,"};\n"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	243
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	244 # FIXME: <fl> the following tables could be made static, and
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	245 # the support code moved into unicodedatabase.c
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	246
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	247 print >>fp, "/* string literals */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	248 print >>fp, "const char *_PyUnicode_CategoryNames[] = {"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	249 for name in CATEGORY_NAMES:
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	250 print >>fp, " \"%s\"," % name
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	251 print >>fp, " NULL"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	252 print >>fp, "};"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	253
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	254 print >>fp, "const char *_PyUnicode_BidirectionalNames[] = {"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	255 for name in BIDIRECTIONAL_NAMES:
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	256 print >>fp, " \"%s\"," % name
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	257 print >>fp, " NULL"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	258 print >>fp, "};"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	259
32951 024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	260 print >>fp, "const char *_PyUnicode_EastAsianWidthNames[] = {"
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	261 for name in EASTASIANWIDTH_NAMES:
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	262 print >>fp, " \"%s\"," % name
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	263 print >>fp, " NULL"
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	264 print >>fp, "};"
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	265
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	266 print >>fp, "static const char *decomp_prefix[] = {"
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	267 for name in decomp_prefix:
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	268 print >>fp, " \"%s\"," % name
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	269 print >>fp, " NULL"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	270 print >>fp, "};"
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	271
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	272 # split record index table
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	273 index1, index2, shift = splitbins(index, trace)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	274
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	275 print >>fp, "/* index tables for the database records */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	276 print >>fp, "#define SHIFT", shift
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	277 Array("index1", index1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	278 Array("index2", index2).dump(fp, trace)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	279
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	280 # split decomposition index table
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	281 index1, index2, shift = splitbins(decomp_index, trace)
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	282
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	283 print >>fp, "/* decomposition data */"
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	284 Array("decomp_data", decomp_data).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	285
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	286 print >>fp, "/* index tables for the decomposition data */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	287 print >>fp, "#define DECOMP_SHIFT", shift
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	288 Array("decomp_index1", index1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	289 Array("decomp_index2", index2).dump(fp, trace)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	290
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	291 index, index2, shift = splitbins(comp_data, trace)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	292 print >>fp, "/* NFC pairs */"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	293 print >>fp, "#define COMP_SHIFT", shift
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	294 Array("comp_index", index).dump(fp, trace)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	295 Array("comp_data", index2).dump(fp, trace)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	296
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	297 # Generate delta tables for old versions
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	298 for version, table, normalization in unicode.changed:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	299 cversion = version.replace(".","_")
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	300 records = [table[0]]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	301 cache = {table[0]:0}
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	302 index = [0] * len(table)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	303 for i, record in enumerate(table):
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	304 try:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	305 index[i] = cache[record]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	306 except KeyError:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	307 index[i] = cache[record] = len(records)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	308 records.append(record)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	309 index1, index2, shift = splitbins(index, trace)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	310 print >>fp, "static const change_record change_records_%s[] = {" % cversion
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	311 for record in records:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	312 print >>fp, "\t{ %s }," % ", ".join(map(str,record))
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	313 print >>fp, "};"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	314 Array("changes_%s_index" % cversion, index1).dump(fp, trace)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	315 Array("changes_%s_data" % cversion, index2).dump(fp, trace)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	316 print >>fp, "static const change_record* get_change_%s(Py_UCS4 n)" % cversion
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	317 print >>fp, "{"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	318 print >>fp, "\tint index;"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	319 print >>fp, "\tif (n >= 0x110000) index = 0;"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	320 print >>fp, "\telse {"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	321 print >>fp, "\t\tindex = changes_%s_index[n>>%d];" % (cversion, shift)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	322 print >>fp, "\t\tindex = changes_%s_data[(index<<%d)+(n & %d)];" % \
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	323 (cversion, shift, ((1<<shift)-1))
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	324 print >>fp, "\t}"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	325 print >>fp, "\treturn change_records_%s+index;" % cversion
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	326 print >>fp, "}\n"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	327 print >>fp, "static Py_UCS4 normalization_%s(Py_UCS4 n)" % cversion
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	328 print >>fp, "{"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	329 print >>fp, "\tswitch(n) {"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	330 for k, v in normalization:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	331 print >>fp, "\tcase %s: return 0x%s;" % (hex(k), v)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	332 print >>fp, "\tdefault: return 0;"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	333 print >>fp, "\t}\n}\n"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	334
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	335 fp.close()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	336
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	337 # --------------------------------------------------------------------
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	338 # unicode character type tables
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	339
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	340 def makeunicodetype(unicode, trace):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	341
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	342 FILE = "Objects/unicodetype_db.h"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	343
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	344 print "--- Preparing", FILE, "..."
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	345
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	346 # extract unicode types
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	347 dummy = (0, 0, 0, 0, 0, 0)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	348 table = [dummy]
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	349 cache = {0: dummy}
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	350 index = [0] * len(unicode.chars)
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	351
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	352 for char in unicode.chars:
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	353 record = unicode.table[char]
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	354 if record:
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	355 # extract database properties
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	356 category = record[2]
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	357 bidirectional = record[4]
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	358 flags = 0
49124 c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	359 delta = True
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	360 if category in ["Lm", "Lt", "Lu", "Ll", "Lo"]:
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	361 flags \|= ALPHA_MASK
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	362 if category == "Ll":
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	363 flags \|= LOWER_MASK
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	364 if category == "Zl" or bidirectional == "B":
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	365 flags \|= LINEBREAK_MASK
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	366 if category == "Zs" or bidirectional in ("WS", "B", "S"):
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	367 flags \|= SPACE_MASK
14856 83aff1ea5b70 - don't set the titlecase flag for uppercase letters (sorry, tim) Fredrik Lundh <fredrik@pythonware.com> parents: 14852 diff changeset	368 if category == "Lt":
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	369 flags \|= TITLE_MASK
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	370 if category == "Lu":
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	371 flags \|= UPPER_MASK
49124 c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	372 # use delta predictor for upper/lower/title if it fits
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	373 if record[12]:
53400 f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	374 upper = int(record[12], 16)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	375 else:
53400 f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	376 upper = char
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	377 if record[13]:
53400 f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	378 lower = int(record[13], 16)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	379 else:
53400 f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	380 lower = char
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	381 if record[14]:
53400 f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	382 title = int(record[14], 16)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	383 else:
53400 f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	384 # UCD.html says that a missing title char means that
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	385 # it defaults to the uppercase character, not to the
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	386 # character itself. Apparently, in the current UCD (5.x)
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	387 # this feature is never used
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	388 title = upper
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	389 upper_d = upper - char
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	390 lower_d = lower - char
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	391 title_d = title - char
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	392 if -32768 <= upper_d <= 32767 and \
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	393 -32768 <= lower_d <= 32767 and \
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	394 -32768 <= title_d <= 32767:
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	395 # use deltas
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	396 upper = upper_d & 0xffff
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	397 lower = lower_d & 0xffff
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	398 title = title_d & 0xffff
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	399 else:
49124 c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	400 flags \|= NODELTA_MASK
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	401 # decimal digit, integer digit
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	402 decimal = 0
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	403 if record[6]:
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	404 flags \|= DECIMAL_MASK
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	405 decimal = int(record[6])
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	406 digit = 0
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	407 if record[7]:
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	408 flags \|= DIGIT_MASK
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	409 digit = int(record[7])
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	410 item = (
32197 3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	411 upper, lower, title, decimal, digit, flags
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	412 )
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	413 # add entry to index and item tables
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	414 i = cache.get(item)
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	415 if i is None:
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	416 cache[item] = i = len(table)
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	417 table.append(item)
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	418 index[char] = i
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	419
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	420 print len(table), "unique character type entries"
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	421
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	422 print "--- Writing", FILE, "..."
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	423
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	424 fp = open(FILE, "w")
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	425 print >>fp, "/* this file was generated by %s %s */" % (SCRIPT, VERSION)
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	426 print >>fp
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	427 print >>fp, "/* a list of unique character type descriptors */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	428 print >>fp, "const _PyUnicode_TypeRecord _PyUnicode_TypeRecords[] = {"
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	429 for item in table:
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	430 print >>fp, " {%d, %d, %d, %d, %d, %d}," % item
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	431 print >>fp, "};"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	432 print >>fp
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	433
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	434 # split decomposition index table
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	435 index1, index2, shift = splitbins(index, trace)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	436
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	437 print >>fp, "/* type indexes */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	438 print >>fp, "#define SHIFT", shift
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	439 Array("index1", index1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	440 Array("index2", index2).dump(fp, trace)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	441
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	442 fp.close()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	443
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	444 # --------------------------------------------------------------------
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	445 # unicode name database
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	446
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	447 def makeunicodename(unicode, trace):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	448
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	449 FILE = "Modules/unicodename_db.h"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	450
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	451 print "--- Preparing", FILE, "..."
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	452
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	453 # collect names
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	454 names = [None] * len(unicode.chars)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	455
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	456 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	457 record = unicode.table[char]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	458 if record:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	459 name = record[1].strip()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	460 if name and name[0] != "<":
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	461 names[char] = name + chr(0)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	462
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	463 print len(filter(lambda n: n is not None, names)), "distinct names"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	464
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	465 # collect unique words from names (note that we differ between
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	466 # words inside a sentence, and words ending a sentence. the
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	467 # latter includes the trailing null byte.
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	468
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	469 words = {}
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	470 n = b = 0
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	471 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	472 name = names[char]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	473 if name:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	474 w = name.split()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	475 b = b + len(name)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	476 n = n + len(w)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	477 for w in w:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	478 l = words.get(w)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	479 if l:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	480 l.append(None)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	481 else:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	482 words[w] = [len(words)]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	483
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	484 print n, "words in text;", b, "bytes"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	485
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	486 wordlist = words.items()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	487
26394 491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	488 # sort on falling frequency, then by name
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	489 def cmpwords((aword, alist),(bword, blist)):
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	490 r = -cmp(len(alist),len(blist))
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	491 if r:
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	492 return r
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	493 return cmp(aword, bword)
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	494 wordlist.sort(cmpwords)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	495
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	496 # figure out how many phrasebook escapes we need
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	497 escapes = 0
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	498 while escapes * 256 < len(wordlist):
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	499 escapes = escapes + 1
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	500 print escapes, "escapes"
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	501
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	502 short = 256 - escapes
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	503
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	504 assert short > 0
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	505
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	506 print short, "short indexes in lexicon"
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	507
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	508 # statistics
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	509 n = 0
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	510 for i in range(short):
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	511 n = n + len(wordlist[i][1])
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	512 print n, "short indexes in phrasebook"
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	513
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	514 # pick the most commonly used words, and sort the rest on falling
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	515 # length (to maximize overlap)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	516
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	517 wordlist, wordtail = wordlist[:short], wordlist[short:]
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	518 wordtail.sort(lambda a, b: len(b[0])-len(a[0]))
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	519 wordlist.extend(wordtail)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	520
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	521 # generate lexicon from words
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	522
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	523 lexicon_offset = [0]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	524 lexicon = ""
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	525 words = {}
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	526
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	527 # build a lexicon string
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	528 offset = 0
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	529 for w, x in wordlist:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	530 # encoding: bit 7 indicates last character in word (chr(128)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	531 # indicates the last character in an entire string)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	532 ww = w[:-1] + chr(ord(w[-1])+128)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	533 # reuse string tails, when possible
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	534 o = lexicon.find(ww)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	535 if o < 0:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	536 o = offset
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	537 lexicon = lexicon + ww
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	538 offset = offset + len(w)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	539 words[w] = len(lexicon_offset)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	540 lexicon_offset.append(o)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	541
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	542 lexicon = map(ord, lexicon)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	543
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	544 # generate phrasebook from names and lexicon
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	545 phrasebook = [0]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	546 phrasebook_offset = [0] * len(unicode.chars)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	547 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	548 name = names[char]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	549 if name:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	550 w = name.split()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	551 phrasebook_offset[char] = len(phrasebook)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	552 for w in w:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	553 i = words[w]
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	554 if i < short:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	555 phrasebook.append(i)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	556 else:
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	557 # store as two bytes
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	558 phrasebook.append((i>>8) + short)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	559 phrasebook.append(i&255)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	560
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	561 assert getsize(phrasebook) == 1
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	562
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	563 #
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	564 # unicode name hash table
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	565
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	566 # extract names
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	567 data = []
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	568 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	569 record = unicode.table[char]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	570 if record:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	571 name = record[1].strip()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	572 if name and name[0] != "<":
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	573 data.append((name, char))
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	574
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	575 # the magic number 47 was chosen to minimize the number of
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	576 # collisions on the current data set. if you like, change it
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	577 # and see what happens...
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	578
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	579 codehash = Hash("code", data, 47)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	580
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	581 print "--- Writing", FILE, "..."
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	582
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	583 fp = open(FILE, "w")
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	584 print >>fp, "/* this file was generated by %s %s */" % (SCRIPT, VERSION)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	585 print >>fp
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	586 print >>fp, "#define NAME_MAXLEN", 256
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	587 print >>fp
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	588 print >>fp, "/* lexicon */"
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	589 Array("lexicon", lexicon).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	590 Array("lexicon_offset", lexicon_offset).dump(fp, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	591
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	592 # split decomposition index table
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	593 offset1, offset2, shift = splitbins(phrasebook_offset, trace)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	594
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	595 print >>fp, "/* code->name phrasebook */"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	596 print >>fp, "#define phrasebook_shift", shift
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	597 print >>fp, "#define phrasebook_short", short
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	598
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	599 Array("phrasebook", phrasebook).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	600 Array("phrasebook_offset1", offset1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	601 Array("phrasebook_offset2", offset2).dump(fp, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	602
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	603 print >>fp, "/* name->code dictionary */"
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	604 codehash.dump(fp, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	605
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	606 fp.close()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	607
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	608
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	609 def merge_old_version(version, new, old):
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	610 # Changes to exclusion file not implemented yet
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	611 if old.exclusions != new.exclusions:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	612 raise NotImplementedError, "exclusions differ"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	613
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	614 # In these change records, 0xFF means "no change"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	615 bidir_changes = [0xFF]*0x110000
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	616 category_changes = [0xFF]*0x110000
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	617 decimal_changes = [0xFF]*0x110000
49124 c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	618 mirrored_changes = [0xFF]*0x110000
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	619 # In numeric data, 0 means "no change",
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	620 # -1 means "did not have a numeric value
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	621 numeric_changes = [0] * 0x110000
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	622 # normalization_changes is a list of key-value pairs
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	623 normalization_changes = []
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	624 for i in range(0x110000):
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	625 if new.table[i] is None:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	626 # Characters unassigned in the new version ought to
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	627 # be unassigned in the old one
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	628 assert old.table[i] is None
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	629 continue
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	630 # check characters unassigned in the old version
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	631 if old.table[i] is None:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	632 # category 0 is "unassigned"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	633 category_changes[i] = 0
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	634 continue
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	635 # check characters that differ
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	636 if old.table[i] != new.table[i]:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	637 for k in range(len(old.table[i])):
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	638 if old.table[i][k] != new.table[i][k]:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	639 value = old.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	640 if k == 2:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	641 #print "CATEGORY",hex(i), old.table[i][k], new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	642 category_changes[i] = CATEGORY_NAMES.index(value)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	643 elif k == 4:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	644 #print "BIDIR",hex(i), old.table[i][k], new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	645 bidir_changes[i] = BIDIRECTIONAL_NAMES.index(value)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	646 elif k == 5:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	647 #print "DECOMP",hex(i), old.table[i][k], new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	648 # We assume that all normalization changes are in 1:1 mappings
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	649 assert " " not in value
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	650 normalization_changes.append((i, value))
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	651 elif k == 6:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	652 #print "DECIMAL",hex(i), old.table[i][k], new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	653 # we only support changes where the old value is a single digit
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	654 assert value in "0123456789"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	655 decimal_changes[i] = int(value)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	656 elif k == 8:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	657 # print "NUMERIC",hex(i), `old.table[i][k]`, new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	658 # Since 0 encodes "no change", the old value is better not 0
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	659 assert value != "0" and value != "-1"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	660 if not value:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	661 numeric_changes[i] = -1
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	662 else:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	663 assert re.match("^[0-9]+$", value)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	664 numeric_changes[i] = int(value)
49124 c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	665 elif k == 9:
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	666 if value == 'Y':
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	667 mirrored_changes[i] = '1'
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	668 else:
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	669 mirrored_changes[i] = '0'
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	670 elif k == 11:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	671 # change to ISO comment, ignore
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	672 pass
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	673 elif k == 12:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	674 # change to simple uppercase mapping; ignore
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	675 pass
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	676 elif k == 13:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	677 # change to simple lowercase mapping; ignore
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	678 pass
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	679 elif k == 14:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	680 # change to simple titlecase mapping; ignore
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	681 pass
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	682 else:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	683 class Difference(Exception):pass
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	684 raise Difference, (hex(i), k, old.table[i], new.table[i])
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	685 new.changed.append((version, zip(bidir_changes, category_changes,
49124 c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	686 decimal_changes, mirrored_changes,
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	687 numeric_changes),
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	688 normalization_changes))
36937 df43c04d2554 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 36924 diff changeset	689
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	690
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	691 # --------------------------------------------------------------------
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	692 # the following support code is taken from the unidb utilities
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	693 # Copyright (c) 1999-2000 by Secret Labs AB
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	694
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	695 # load a unicode-data file from disk
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	696
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	697 import sys
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	698
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	699 class UnicodeData:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	700
32197 3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	701 def __init__(self, filename, exclusions, eastasianwidth, expand=1):
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	702 self.changed = []
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	703 file = open(filename)
25852 1f5f6481886d Update to Unicode 3.2 database. Martin v. Löwis <martin@v.loewis.de> parents: 25219 diff changeset	704 table = [None] * 0x110000
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	705 while 1:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	706 s = file.readline()
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	707 if not s:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	708 break
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	709 s = s.strip().split(";")
8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	710 char = int(s[0], 16)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	711 table[char] = s
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	712
26394 491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	713 # expand first-last ranges
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	714 if expand:
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	715 field = None
26394 491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	716 for i in range(0, 0x110000):
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	717 s = table[i]
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	718 if s:
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	719 if s[1][-6:] == "First>":
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	720 s[1] = ""
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	721 field = s
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	722 elif s[1][-5:] == "Last>":
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	723 s[1] = ""
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	724 field = None
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	725 elif field:
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	726 f2 = field[:]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	727 f2[0] = "%X" % i
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	728 table[i] = f2
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	729
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	730 # public attributes
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	731 self.filename = filename
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	732 self.table = table
25852 1f5f6481886d Update to Unicode 3.2 database. Martin v. Löwis <martin@v.loewis.de> parents: 25219 diff changeset	733 self.chars = range(0x110000) # unicode 3.2
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	734
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	735 file = open(exclusions)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	736 self.exclusions = {}
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	737 for s in file:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	738 s = s.strip()
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	739 if not s:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	740 continue
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	741 if s[0] == '#':
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	742 continue
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	743 char = int(s.split()[0],16)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	744 self.exclusions[char] = 1
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	745
32197 3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	746 widths = [None] * 0x110000
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	747 for s in open(eastasianwidth):
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	748 s = s.strip()
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	749 if not s:
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	750 continue
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	751 if s[0] == '#':
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	752 continue
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	753 s = s.split()[0].split(';')
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	754 if '..' in s[0]:
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	755 first, last = [int(c, 16) for c in s[0].split('..')]
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	756 chars = range(first, last+1)
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	757 else:
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	758 chars = [int(s[0], 16)]
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	759 for char in chars:
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	760 widths[char] = s[1]
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	761 for i in range(0, 0x110000):
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	762 if table[i] is not None:
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	763 table[i].append(widths[i])
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	764
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	765 def uselatin1(self):
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	766 # restrict character range to ISO Latin 1
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	767 self.chars = range(256)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	768
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	769 # hash table tools
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	770
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	771 # this is a straight-forward reimplementation of Python's built-in
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	772 # dictionary type, using a static data structure, and a custom string
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	773 # hash algorithm.
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	774
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	775 def myhash(s, magic):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	776 h = 0
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	777 for c in map(ord, s.upper()):
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	778 h = (h * magic) + c
26394 491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	779 ix = h & 0xff000000L
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	780 if ix:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	781 h = (h ^ ((ix>>24) & 0xff)) & 0x00ffffff
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	782 return h
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	783
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	784 SIZES = [
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	785 (4,3), (8,3), (16,3), (32,5), (64,3), (128,3), (256,29), (512,17),
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	786 (1024,9), (2048,5), (4096,83), (8192,27), (16384,43), (32768,3),
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	787 (65536,45), (131072,9), (262144,39), (524288,39), (1048576,9),
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	788 (2097152,5), (4194304,3), (8388608,33), (16777216,27)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	789 ]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	790
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	791 class Hash:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	792 def __init__(self, name, data, magic):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	793 # turn a (key, value) list into a static hash table structure
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	794
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	795 # determine table size
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	796 for size, poly in SIZES:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	797 if size > len(data):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	798 poly = size + poly
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	799 break
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	800 else:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	801 raise AssertionError, "ran out of polynominals"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	802
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	803 print size, "slots in hash table"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	804
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	805 table = [None] * size
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	806
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	807 mask = size-1
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	808
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	809 n = 0
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	810
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	811 hash = myhash
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	812
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	813 # initialize hash table
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	814 for key, value in data:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	815 h = hash(key, magic)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	816 i = (~h) & mask
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	817 v = table[i]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	818 if v is None:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	819 table[i] = value
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	820 continue
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	821 incr = (h ^ (h >> 3)) & mask;
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	822 if not incr:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	823 incr = mask
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	824 while 1:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	825 n = n + 1
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	826 i = (i + incr) & mask
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	827 v = table[i]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	828 if v is None:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	829 table[i] = value
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	830 break
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	831 incr = incr << 1
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	832 if incr > mask:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	833 incr = incr ^ poly
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	834
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	835 print n, "collisions"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	836 self.collisions = n
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	837
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	838 for i in range(len(table)):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	839 if table[i] is None:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	840 table[i] = 0
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	841
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	842 self.data = Array(name + "_hash", table)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	843 self.magic = magic
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	844 self.name = name
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	845 self.size = size
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	846 self.poly = poly
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	847
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	848 def dump(self, file, trace):
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	849 # write data to file, as a C array
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	850 self.data.dump(file, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	851 file.write("#define %s_magic %d\n" % (self.name, self.magic))
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	852 file.write("#define %s_size %d\n" % (self.name, self.size))
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	853 file.write("#define %s_poly %d\n" % (self.name, self.poly))
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	854
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	855 # stuff to deal with arrays of unsigned integers
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	856
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	857 class Array:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	858
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	859 def __init__(self, name, data):
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	860 self.name = name
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	861 self.data = data
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	862
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	863 def dump(self, file, trace=0):
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	864 # write data to file, as a C array
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	865 size = getsize(self.data)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	866 if trace:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	867 print >>sys.stderr, self.name+":", size*len(self.data), "bytes"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	868 file.write("static ")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	869 if size == 1:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	870 file.write("unsigned char")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	871 elif size == 2:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	872 file.write("unsigned short")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	873 else:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	874 file.write("unsigned int")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	875 file.write(" " + self.name + "[] = {\n")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	876 if self.data:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	877 s = " "
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	878 for item in self.data:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	879 i = str(item) + ", "
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	880 if len(s) + len(i) > 78:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	881 file.write(s + "\n")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	882 s = " " + i
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	883 else:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	884 s = s + i
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	885 if s.strip():
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	886 file.write(s + "\n")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	887 file.write("};\n\n")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	888
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	889 def getsize(data):
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	890 # return smallest possible integer size for the given array
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	891 maxdata = max(data)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	892 if maxdata < 256:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	893 return 1
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	894 elif maxdata < 65536:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	895 return 2
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	896 else:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	897 return 4
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	898
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	899 def splitbins(t, trace=0):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	900 """t, trace=0 -> (t1, t2, shift). Split a table to save space.
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	901
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	902 t is a sequence of ints. This function can be useful to save space if
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	903 many of the ints are the same. t1 and t2 are lists of ints, and shift
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	904 is an int, chosen to minimize the combined size of t1 and t2 (in C
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	905 code), and where for each i in range(len(t)),
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	906 t[i] == t2[(t1[i >> shift] << shift) + (i & mask)]
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	907 where mask is a bitmask isolating the last "shift" bits.
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	908
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	909 If optional arg trace is non-zero (default zero), progress info
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	910 is printed to sys.stderr. The higher the value, the more info
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	911 you'll get.
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	912 """
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	913
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	914 import sys
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	915 if trace:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	916 def dump(t1, t2, shift, bytes):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	917 print >>sys.stderr, "%d+%d bins at shift %d; %d bytes" % (
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	918 len(t1), len(t2), shift, bytes)
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	919 print >>sys.stderr, "Size of original table:", len(t)*getsize(t), \
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	920 "bytes"
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	921 n = len(t)-1 # last valid index
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	922 maxshift = 0 # the most we can shift n and still have something left
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	923 if n > 0:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	924 while n >> 1:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	925 n >>= 1
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	926 maxshift += 1
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	927 del n
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	928 bytes = sys.maxint # smallest total size so far
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	929 t = tuple(t) # so slices can be dict keys
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	930 for shift in range(maxshift + 1):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	931 t1 = []
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	932 t2 = []
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	933 size = 2**shift
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	934 bincache = {}
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	935 for i in range(0, len(t), size):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	936 bin = t[i:i+size]
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	937 index = bincache.get(bin)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	938 if index is None:
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	939 index = len(t2)
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	940 bincache[bin] = index
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	941 t2.extend(bin)
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	942 t1.append(index >> shift)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	943 # determine memory size
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	944 b = len(t1)getsize(t1) + len(t2)getsize(t2)
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	945 if trace > 1:
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	946 dump(t1, t2, shift, b)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	947 if b < bytes:
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	948 best = t1, t2, shift
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	949 bytes = b
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	950 t1, t2, shift = best
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	951 if trace:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	952 print >>sys.stderr, "Best:",
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	953 dump(t1, t2, shift, bytes)
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	954 if __debug__:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	955 # exhaustively verify that the decomposition is correct
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	956 mask = ~((~0) << shift) # i.e., low-bit mask of shift bits
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	957 for i in xrange(len(t)):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	958 assert t[i] == t2[(t1[i >> shift] << shift) + (i & mask)]
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	959 return best
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	960
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	961 if __name__ == "__main__":
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	962 maketables(1)

Mercurial > cpython-lru_cache-weakref

annotate Tools/unicode/makeunicodedata.py @ 58925:eecf1bef052d 2.6