cpython-lru_cache-weakref: Tools/unicode/makeunicodedata.py annotate

annotate Tools/unicode/makeunicodedata.py @ 54294:dba767bc3155 legacy-trunk

remove mention of old ctypes version

author	Benjamin Peterson <benjamin@python.org>
date	Sat, 23 May 2009 20:59:09 +0000
parents	019244f4b0d0
children	a9864417286b

rev	line source
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1 #
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	2 # (re)generate unicode property and type databases
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	3 #
26397 b27c1cad77ea Add unidata_version. Bump generator version number. Martin v. Löwis <martin@v.loewis.de> parents: 26394 diff changeset	4 # this script converts a unicode 3.2 database file to
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	5 # Modules/unicodedata_db.h, Modules/unicodename_db.h,
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	6 # and Objects/unicodetype_db.h
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	7 #
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	8 # history:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	9 # 2000-09-24 fl created (based on bits and pieces from unidb)
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	10 # 2000-09-25 fl merged tim's splitbin fixes, separate decomposition table
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	11 # 2000-09-25 fl added character type table
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	12 # 2000-09-26 fl added LINEBREAK, DECIMAL, and DIGIT flags/fields (2.0)
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	13 # 2000-11-03 fl expand first/last ranges
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	14 # 2001-01-19 fl added character name tables (2.1)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	15 # 2001-01-21 fl added decomp compression; dynamic phrasebook threshold
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	16 # 2002-09-11 wd use string methods
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	17 # 2002-10-18 mvl update to Unicode 3.2
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	18 # 2002-10-22 mvl generate NFC tables
26394 491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	19 # 2002-11-24 mvl expand all ranges, sort names version-independently
26397 b27c1cad77ea Add unidata_version. Bump generator version number. Martin v. Löwis <martin@v.loewis.de> parents: 26394 diff changeset	20 # 2002-11-25 mvl add UNIDATA_VERSION
32197 3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	21 # 2004-05-29 perky add east asian width information
36940 e4465d2737c4 Add changelog entry. Martin v. Löwis <martin@v.loewis.de> parents: 36937 diff changeset	22 # 2006-03-10 mvl update to Unicode 4.1; add UCD 3.2 delta
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	23 #
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	24 # written by Fredrik Lundh (fredrik@pythonware.com)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	25 #
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	26
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	27 import sys
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	28
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	29 SCRIPT = sys.argv[0]
49124 c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	30 VERSION = "2.6"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	31
26397 b27c1cad77ea Add unidata_version. Bump generator version number. Martin v. Löwis <martin@v.loewis.de> parents: 26394 diff changeset	32 # The Unicode Database
49124 c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	33 UNIDATA_VERSION = "5.1.0"
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	34 UNICODE_DATA = "UnicodeData%s.txt"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	35 COMPOSITION_EXCLUSIONS = "CompositionExclusions%s.txt"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	36 EASTASIAN_WIDTH = "EastAsianWidth%s.txt"
53553 019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	37 DERIVEDNORMALIZATION_PROPS = "DerivedNormalizationProps%s.txt"
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	38
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	39 old_versions = ["3.2.0"]
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	40
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	41 CATEGORY_NAMES = [ "Cn", "Lu", "Ll", "Lt", "Mn", "Mc", "Me", "Nd",
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	42 "Nl", "No", "Zs", "Zl", "Zp", "Cc", "Cf", "Cs", "Co", "Cn", "Lm",
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	43 "Lo", "Pc", "Pd", "Ps", "Pe", "Pi", "Pf", "Po", "Sm", "Sc", "Sk",
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	44 "So" ]
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	45
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	46 BIDIRECTIONAL_NAMES = [ "", "L", "LRE", "LRO", "R", "AL", "RLE", "RLO",
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	47 "PDF", "EN", "ES", "ET", "AN", "CS", "NSM", "BN", "B", "S", "WS",
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	48 "ON" ]
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	49
32951 024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	50 EASTASIANWIDTH_NAMES = [ "F", "H", "W", "Na", "A", "N" ]
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	51
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	52 # note: should match definitions in Objects/unicodectype.c
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	53 ALPHA_MASK = 0x01
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	54 DECIMAL_MASK = 0x02
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	55 DIGIT_MASK = 0x04
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	56 LOWER_MASK = 0x08
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	57 LINEBREAK_MASK = 0x10
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	58 SPACE_MASK = 0x20
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	59 TITLE_MASK = 0x40
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	60 UPPER_MASK = 0x80
49124 c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	61 NODELTA_MASK = 0x100
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	62
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	63 def maketables(trace=0):
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	64
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	65 print "--- Reading", UNICODE_DATA % "", "..."
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	66
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	67 version = ""
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	68 unicode = UnicodeData(UNICODE_DATA % version,
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	69 COMPOSITION_EXCLUSIONS % version,
53553 019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	70 EASTASIAN_WIDTH % version,
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	71 DERIVEDNORMALIZATION_PROPS % version)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	72
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	73 print len(filter(None, unicode.table)), "characters"
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	74
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	75 for version in old_versions:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	76 print "--- Reading", UNICODE_DATA % ("-"+version), "..."
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	77 old_unicode = UnicodeData(UNICODE_DATA % ("-"+version),
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	78 COMPOSITION_EXCLUSIONS % ("-"+version),
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	79 EASTASIAN_WIDTH % ("-"+version))
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	80 print len(filter(None, old_unicode.table)), "characters"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	81 merge_old_version(version, unicode, old_unicode)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	82
16091 45c065668ea5 Unicode nits: Don't include unicodedatabase.h no more. And make sure Fredrik Lundh <fredrik@pythonware.com> parents: 16090 diff changeset	83 makeunicodename(unicode, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	84 makeunicodedata(unicode, trace)
16091 45c065668ea5 Unicode nits: Don't include unicodedatabase.h no more. And make sure Fredrik Lundh <fredrik@pythonware.com> parents: 16090 diff changeset	85 makeunicodetype(unicode, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	86
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	87 # --------------------------------------------------------------------
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	88 # unicode character properties
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	89
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	90 def makeunicodedata(unicode, trace):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	91
53553 019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	92 dummy = (0, 0, 0, 0, 0, 0)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	93 table = [dummy]
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	94 cache = {0: dummy}
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	95 index = [0] * len(unicode.chars)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	96
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	97 FILE = "Modules/unicodedata_db.h"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	98
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	99 print "--- Preparing", FILE, "..."
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	100
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	101 # 1) database properties
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	102
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	103 for char in unicode.chars:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	104 record = unicode.table[char]
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	105 if record:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	106 # extract database properties
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	107 category = CATEGORY_NAMES.index(record[2])
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	108 combining = int(record[3])
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	109 bidirectional = BIDIRECTIONAL_NAMES.index(record[4])
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	110 mirrored = record[9] == "Y"
32951 024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	111 eastasianwidth = EASTASIANWIDTH_NAMES.index(record[15])
53553 019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	112 normalizationquickcheck = record[16]
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	113 item = (
53553 019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	114 category, combining, bidirectional, mirrored, eastasianwidth,
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	115 normalizationquickcheck
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	116 )
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	117 # add entry to index and item tables
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	118 i = cache.get(item)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	119 if i is None:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	120 cache[item] = i = len(table)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	121 table.append(item)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	122 index[char] = i
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	123
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	124 # 2) decomposition data
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	125
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	126 decomp_data = [0]
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	127 decomp_prefix = [""]
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	128 decomp_index = [0] * len(unicode.chars)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	129 decomp_size = 0
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	130
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	131 comp_pairs = []
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	132 comp_first = [None] * len(unicode.chars)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	133 comp_last = [None] * len(unicode.chars)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	134
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	135 for char in unicode.chars:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	136 record = unicode.table[char]
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	137 if record:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	138 if record[5]:
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	139 decomp = record[5].split()
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	140 if len(decomp) > 19:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	141 raise Exception, "character %x has a decomposition too large for nfd_nfkd" % char
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	142 # prefix
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	143 if decomp[0][0] == "<":
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	144 prefix = decomp.pop(0)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	145 else:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	146 prefix = ""
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	147 try:
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	148 i = decomp_prefix.index(prefix)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	149 except ValueError:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	150 i = len(decomp_prefix)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	151 decomp_prefix.append(prefix)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	152 prefix = i
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	153 assert prefix < 256
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	154 # content
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	155 decomp = [prefix + (len(decomp)<<8)] +\
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	156 map(lambda s: int(s, 16), decomp)
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	157 # Collect NFC pairs
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	158 if not prefix and len(decomp) == 3 and \
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	159 char not in unicode.exclusions and \
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	160 unicode.table[decomp[1]][3] == "0":
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	161 p, l, r = decomp
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	162 comp_first[l] = 1
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	163 comp_last[r] = 1
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	164 comp_pairs.append((l,r,char))
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	165 try:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	166 i = decomp_data.index(decomp)
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	167 except ValueError:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	168 i = len(decomp_data)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	169 decomp_data.extend(decomp)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	170 decomp_size = decomp_size + len(decomp) * 2
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	171 else:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	172 i = 0
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	173 decomp_index[char] = i
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	174
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	175 f = l = 0
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	176 comp_first_ranges = []
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	177 comp_last_ranges = []
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	178 prev_f = prev_l = None
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	179 for i in unicode.chars:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	180 if comp_first[i] is not None:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	181 comp_first[i] = f
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	182 f += 1
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	183 if prev_f is None:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	184 prev_f = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	185 elif prev_f[1]+1 == i:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	186 prev_f = prev_f[0],i
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	187 else:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	188 comp_first_ranges.append(prev_f)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	189 prev_f = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	190 if comp_last[i] is not None:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	191 comp_last[i] = l
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	192 l += 1
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	193 if prev_l is None:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	194 prev_l = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	195 elif prev_l[1]+1 == i:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	196 prev_l = prev_l[0],i
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	197 else:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	198 comp_last_ranges.append(prev_l)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	199 prev_l = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	200 comp_first_ranges.append(prev_f)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	201 comp_last_ranges.append(prev_l)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	202 total_first = f
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	203 total_last = l
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	204
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	205 comp_data = [0](total_firsttotal_last)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	206 for f,l,char in comp_pairs:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	207 f = comp_first[f]
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	208 l = comp_last[l]
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	209 comp_data[f*total_last+l] = char
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	210
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	211 print len(table), "unique properties"
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	212 print len(decomp_prefix), "unique decomposition prefixes"
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	213 print len(decomp_data), "unique decomposition entries:",
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	214 print decomp_size, "bytes"
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	215 print total_first, "first characters in NFC"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	216 print total_last, "last characters in NFC"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	217 print len(comp_pairs), "NFC pairs"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	218
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	219 print "--- Writing", FILE, "..."
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	220
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	221 fp = open(FILE, "w")
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	222 print >>fp, "/* this file was generated by %s %s */" % (SCRIPT, VERSION)
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	223 print >>fp
26397 b27c1cad77ea Add unidata_version. Bump generator version number. Martin v. Löwis <martin@v.loewis.de> parents: 26394 diff changeset	224 print >>fp, '#define UNIDATA_VERSION "%s"' % UNIDATA_VERSION
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	225 print >>fp, "/* a list of unique database records */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	226 print >>fp, \
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	227 "const _PyUnicode_DatabaseRecord _PyUnicode_Database_Records[] = {"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	228 for item in table:
53553 019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	229 print >>fp, " {%d, %d, %d, %d, %d, %d}," % item
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	230 print >>fp, "};"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	231 print >>fp
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	232
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	233 print >>fp, "/* Reindexing of NFC first characters. */"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	234 print >>fp, "#define TOTAL_FIRST",total_first
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	235 print >>fp, "#define TOTAL_LAST",total_last
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	236 print >>fp, "struct reindex{int start;short count,index;};"
47678 a35da94e05fc Make more symbols static. Martin v. Löwis <martin@v.loewis.de> parents: 36940 diff changeset	237 print >>fp, "static struct reindex nfc_first[] = {"
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	238 for start,end in comp_first_ranges:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	239 print >>fp," { %d, %d, %d}," % (start,end-start,comp_first[start])
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	240 print >>fp," {0,0,0}"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	241 print >>fp,"};\n"
47678 a35da94e05fc Make more symbols static. Martin v. Löwis <martin@v.loewis.de> parents: 36940 diff changeset	242 print >>fp, "static struct reindex nfc_last[] = {"
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	243 for start,end in comp_last_ranges:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	244 print >>fp," { %d, %d, %d}," % (start,end-start,comp_last[start])
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	245 print >>fp," {0,0,0}"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	246 print >>fp,"};\n"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	247
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	248 # FIXME: <fl> the following tables could be made static, and
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	249 # the support code moved into unicodedatabase.c
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	250
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	251 print >>fp, "/* string literals */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	252 print >>fp, "const char *_PyUnicode_CategoryNames[] = {"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	253 for name in CATEGORY_NAMES:
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	254 print >>fp, " \"%s\"," % name
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	255 print >>fp, " NULL"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	256 print >>fp, "};"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	257
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	258 print >>fp, "const char *_PyUnicode_BidirectionalNames[] = {"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	259 for name in BIDIRECTIONAL_NAMES:
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	260 print >>fp, " \"%s\"," % name
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	261 print >>fp, " NULL"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	262 print >>fp, "};"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	263
32951 024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	264 print >>fp, "const char *_PyUnicode_EastAsianWidthNames[] = {"
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	265 for name in EASTASIANWIDTH_NAMES:
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	266 print >>fp, " \"%s\"," % name
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	267 print >>fp, " NULL"
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	268 print >>fp, "};"
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	269
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	270 print >>fp, "static const char *decomp_prefix[] = {"
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	271 for name in decomp_prefix:
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	272 print >>fp, " \"%s\"," % name
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	273 print >>fp, " NULL"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	274 print >>fp, "};"
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	275
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	276 # split record index table
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	277 index1, index2, shift = splitbins(index, trace)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	278
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	279 print >>fp, "/* index tables for the database records */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	280 print >>fp, "#define SHIFT", shift
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	281 Array("index1", index1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	282 Array("index2", index2).dump(fp, trace)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	283
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	284 # split decomposition index table
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	285 index1, index2, shift = splitbins(decomp_index, trace)
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	286
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	287 print >>fp, "/* decomposition data */"
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	288 Array("decomp_data", decomp_data).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	289
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	290 print >>fp, "/* index tables for the decomposition data */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	291 print >>fp, "#define DECOMP_SHIFT", shift
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	292 Array("decomp_index1", index1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	293 Array("decomp_index2", index2).dump(fp, trace)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	294
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	295 index, index2, shift = splitbins(comp_data, trace)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	296 print >>fp, "/* NFC pairs */"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	297 print >>fp, "#define COMP_SHIFT", shift
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	298 Array("comp_index", index).dump(fp, trace)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	299 Array("comp_data", index2).dump(fp, trace)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	300
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	301 # Generate delta tables for old versions
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	302 for version, table, normalization in unicode.changed:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	303 cversion = version.replace(".","_")
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	304 records = [table[0]]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	305 cache = {table[0]:0}
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	306 index = [0] * len(table)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	307 for i, record in enumerate(table):
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	308 try:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	309 index[i] = cache[record]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	310 except KeyError:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	311 index[i] = cache[record] = len(records)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	312 records.append(record)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	313 index1, index2, shift = splitbins(index, trace)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	314 print >>fp, "static const change_record change_records_%s[] = {" % cversion
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	315 for record in records:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	316 print >>fp, "\t{ %s }," % ", ".join(map(str,record))
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	317 print >>fp, "};"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	318 Array("changes_%s_index" % cversion, index1).dump(fp, trace)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	319 Array("changes_%s_data" % cversion, index2).dump(fp, trace)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	320 print >>fp, "static const change_record* get_change_%s(Py_UCS4 n)" % cversion
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	321 print >>fp, "{"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	322 print >>fp, "\tint index;"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	323 print >>fp, "\tif (n >= 0x110000) index = 0;"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	324 print >>fp, "\telse {"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	325 print >>fp, "\t\tindex = changes_%s_index[n>>%d];" % (cversion, shift)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	326 print >>fp, "\t\tindex = changes_%s_data[(index<<%d)+(n & %d)];" % \
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	327 (cversion, shift, ((1<<shift)-1))
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	328 print >>fp, "\t}"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	329 print >>fp, "\treturn change_records_%s+index;" % cversion
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	330 print >>fp, "}\n"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	331 print >>fp, "static Py_UCS4 normalization_%s(Py_UCS4 n)" % cversion
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	332 print >>fp, "{"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	333 print >>fp, "\tswitch(n) {"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	334 for k, v in normalization:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	335 print >>fp, "\tcase %s: return 0x%s;" % (hex(k), v)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	336 print >>fp, "\tdefault: return 0;"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	337 print >>fp, "\t}\n}\n"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	338
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	339 fp.close()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	340
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	341 # --------------------------------------------------------------------
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	342 # unicode character type tables
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	343
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	344 def makeunicodetype(unicode, trace):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	345
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	346 FILE = "Objects/unicodetype_db.h"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	347
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	348 print "--- Preparing", FILE, "..."
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	349
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	350 # extract unicode types
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	351 dummy = (0, 0, 0, 0, 0, 0)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	352 table = [dummy]
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	353 cache = {0: dummy}
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	354 index = [0] * len(unicode.chars)
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	355
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	356 for char in unicode.chars:
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	357 record = unicode.table[char]
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	358 if record:
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	359 # extract database properties
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	360 category = record[2]
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	361 bidirectional = record[4]
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	362 flags = 0
49124 c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	363 delta = True
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	364 if category in ["Lm", "Lt", "Lu", "Ll", "Lo"]:
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	365 flags \|= ALPHA_MASK
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	366 if category == "Ll":
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	367 flags \|= LOWER_MASK
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	368 if category == "Zl" or bidirectional == "B":
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	369 flags \|= LINEBREAK_MASK
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	370 if category == "Zs" or bidirectional in ("WS", "B", "S"):
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	371 flags \|= SPACE_MASK
14856 83aff1ea5b70 - don't set the titlecase flag for uppercase letters (sorry, tim) Fredrik Lundh <fredrik@pythonware.com> parents: 14852 diff changeset	372 if category == "Lt":
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	373 flags \|= TITLE_MASK
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	374 if category == "Lu":
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	375 flags \|= UPPER_MASK
49124 c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	376 # use delta predictor for upper/lower/title if it fits
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	377 if record[12]:
53400 f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	378 upper = int(record[12], 16)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	379 else:
53400 f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	380 upper = char
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	381 if record[13]:
53400 f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	382 lower = int(record[13], 16)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	383 else:
53400 f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	384 lower = char
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	385 if record[14]:
53400 f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	386 title = int(record[14], 16)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	387 else:
53400 f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	388 # UCD.html says that a missing title char means that
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	389 # it defaults to the uppercase character, not to the
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	390 # character itself. Apparently, in the current UCD (5.x)
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	391 # this feature is never used
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	392 title = upper
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	393 upper_d = upper - char
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	394 lower_d = lower - char
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	395 title_d = title - char
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	396 if -32768 <= upper_d <= 32767 and \
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	397 -32768 <= lower_d <= 32767 and \
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	398 -32768 <= title_d <= 32767:
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	399 # use deltas
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	400 upper = upper_d & 0xffff
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	401 lower = lower_d & 0xffff
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	402 title = title_d & 0xffff
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	403 else:
49124 c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	404 flags \|= NODELTA_MASK
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	405 # decimal digit, integer digit
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	406 decimal = 0
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	407 if record[6]:
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	408 flags \|= DECIMAL_MASK
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	409 decimal = int(record[6])
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	410 digit = 0
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	411 if record[7]:
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	412 flags \|= DIGIT_MASK
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	413 digit = int(record[7])
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	414 item = (
32197 3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	415 upper, lower, title, decimal, digit, flags
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	416 )
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	417 # add entry to index and item tables
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	418 i = cache.get(item)
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	419 if i is None:
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	420 cache[item] = i = len(table)
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	421 table.append(item)
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	422 index[char] = i
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	423
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	424 print len(table), "unique character type entries"
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	425
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	426 print "--- Writing", FILE, "..."
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	427
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	428 fp = open(FILE, "w")
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	429 print >>fp, "/* this file was generated by %s %s */" % (SCRIPT, VERSION)
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	430 print >>fp
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	431 print >>fp, "/* a list of unique character type descriptors */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	432 print >>fp, "const _PyUnicode_TypeRecord _PyUnicode_TypeRecords[] = {"
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	433 for item in table:
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	434 print >>fp, " {%d, %d, %d, %d, %d, %d}," % item
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	435 print >>fp, "};"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	436 print >>fp
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	437
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	438 # split decomposition index table
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	439 index1, index2, shift = splitbins(index, trace)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	440
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	441 print >>fp, "/* type indexes */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	442 print >>fp, "#define SHIFT", shift
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	443 Array("index1", index1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	444 Array("index2", index2).dump(fp, trace)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	445
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	446 fp.close()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	447
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	448 # --------------------------------------------------------------------
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	449 # unicode name database
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	450
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	451 def makeunicodename(unicode, trace):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	452
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	453 FILE = "Modules/unicodename_db.h"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	454
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	455 print "--- Preparing", FILE, "..."
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	456
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	457 # collect names
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	458 names = [None] * len(unicode.chars)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	459
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	460 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	461 record = unicode.table[char]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	462 if record:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	463 name = record[1].strip()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	464 if name and name[0] != "<":
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	465 names[char] = name + chr(0)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	466
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	467 print len(filter(lambda n: n is not None, names)), "distinct names"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	468
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	469 # collect unique words from names (note that we differ between
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	470 # words inside a sentence, and words ending a sentence. the
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	471 # latter includes the trailing null byte.
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	472
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	473 words = {}
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	474 n = b = 0
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	475 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	476 name = names[char]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	477 if name:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	478 w = name.split()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	479 b = b + len(name)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	480 n = n + len(w)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	481 for w in w:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	482 l = words.get(w)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	483 if l:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	484 l.append(None)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	485 else:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	486 words[w] = [len(words)]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	487
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	488 print n, "words in text;", b, "bytes"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	489
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	490 wordlist = words.items()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	491
26394 491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	492 # sort on falling frequency, then by name
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	493 def cmpwords((aword, alist),(bword, blist)):
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	494 r = -cmp(len(alist),len(blist))
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	495 if r:
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	496 return r
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	497 return cmp(aword, bword)
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	498 wordlist.sort(cmpwords)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	499
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	500 # figure out how many phrasebook escapes we need
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	501 escapes = 0
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	502 while escapes * 256 < len(wordlist):
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	503 escapes = escapes + 1
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	504 print escapes, "escapes"
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	505
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	506 short = 256 - escapes
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	507
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	508 assert short > 0
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	509
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	510 print short, "short indexes in lexicon"
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	511
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	512 # statistics
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	513 n = 0
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	514 for i in range(short):
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	515 n = n + len(wordlist[i][1])
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	516 print n, "short indexes in phrasebook"
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	517
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	518 # pick the most commonly used words, and sort the rest on falling
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	519 # length (to maximize overlap)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	520
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	521 wordlist, wordtail = wordlist[:short], wordlist[short:]
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	522 wordtail.sort(lambda a, b: len(b[0])-len(a[0]))
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	523 wordlist.extend(wordtail)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	524
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	525 # generate lexicon from words
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	526
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	527 lexicon_offset = [0]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	528 lexicon = ""
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	529 words = {}
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	530
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	531 # build a lexicon string
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	532 offset = 0
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	533 for w, x in wordlist:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	534 # encoding: bit 7 indicates last character in word (chr(128)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	535 # indicates the last character in an entire string)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	536 ww = w[:-1] + chr(ord(w[-1])+128)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	537 # reuse string tails, when possible
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	538 o = lexicon.find(ww)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	539 if o < 0:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	540 o = offset
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	541 lexicon = lexicon + ww
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	542 offset = offset + len(w)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	543 words[w] = len(lexicon_offset)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	544 lexicon_offset.append(o)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	545
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	546 lexicon = map(ord, lexicon)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	547
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	548 # generate phrasebook from names and lexicon
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	549 phrasebook = [0]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	550 phrasebook_offset = [0] * len(unicode.chars)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	551 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	552 name = names[char]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	553 if name:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	554 w = name.split()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	555 phrasebook_offset[char] = len(phrasebook)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	556 for w in w:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	557 i = words[w]
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	558 if i < short:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	559 phrasebook.append(i)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	560 else:
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	561 # store as two bytes
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	562 phrasebook.append((i>>8) + short)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	563 phrasebook.append(i&255)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	564
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	565 assert getsize(phrasebook) == 1
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	566
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	567 #
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	568 # unicode name hash table
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	569
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	570 # extract names
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	571 data = []
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	572 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	573 record = unicode.table[char]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	574 if record:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	575 name = record[1].strip()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	576 if name and name[0] != "<":
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	577 data.append((name, char))
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	578
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	579 # the magic number 47 was chosen to minimize the number of
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	580 # collisions on the current data set. if you like, change it
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	581 # and see what happens...
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	582
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	583 codehash = Hash("code", data, 47)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	584
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	585 print "--- Writing", FILE, "..."
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	586
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	587 fp = open(FILE, "w")
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	588 print >>fp, "/* this file was generated by %s %s */" % (SCRIPT, VERSION)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	589 print >>fp
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	590 print >>fp, "#define NAME_MAXLEN", 256
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	591 print >>fp
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	592 print >>fp, "/* lexicon */"
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	593 Array("lexicon", lexicon).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	594 Array("lexicon_offset", lexicon_offset).dump(fp, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	595
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	596 # split decomposition index table
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	597 offset1, offset2, shift = splitbins(phrasebook_offset, trace)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	598
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	599 print >>fp, "/* code->name phrasebook */"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	600 print >>fp, "#define phrasebook_shift", shift
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	601 print >>fp, "#define phrasebook_short", short
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	602
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	603 Array("phrasebook", phrasebook).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	604 Array("phrasebook_offset1", offset1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	605 Array("phrasebook_offset2", offset2).dump(fp, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	606
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	607 print >>fp, "/* name->code dictionary */"
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	608 codehash.dump(fp, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	609
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	610 fp.close()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	611
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	612
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	613 def merge_old_version(version, new, old):
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	614 # Changes to exclusion file not implemented yet
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	615 if old.exclusions != new.exclusions:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	616 raise NotImplementedError, "exclusions differ"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	617
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	618 # In these change records, 0xFF means "no change"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	619 bidir_changes = [0xFF]*0x110000
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	620 category_changes = [0xFF]*0x110000
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	621 decimal_changes = [0xFF]*0x110000
49124 c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	622 mirrored_changes = [0xFF]*0x110000
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	623 # In numeric data, 0 means "no change",
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	624 # -1 means "did not have a numeric value
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	625 numeric_changes = [0] * 0x110000
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	626 # normalization_changes is a list of key-value pairs
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	627 normalization_changes = []
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	628 for i in range(0x110000):
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	629 if new.table[i] is None:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	630 # Characters unassigned in the new version ought to
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	631 # be unassigned in the old one
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	632 assert old.table[i] is None
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	633 continue
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	634 # check characters unassigned in the old version
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	635 if old.table[i] is None:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	636 # category 0 is "unassigned"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	637 category_changes[i] = 0
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	638 continue
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	639 # check characters that differ
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	640 if old.table[i] != new.table[i]:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	641 for k in range(len(old.table[i])):
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	642 if old.table[i][k] != new.table[i][k]:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	643 value = old.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	644 if k == 2:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	645 #print "CATEGORY",hex(i), old.table[i][k], new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	646 category_changes[i] = CATEGORY_NAMES.index(value)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	647 elif k == 4:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	648 #print "BIDIR",hex(i), old.table[i][k], new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	649 bidir_changes[i] = BIDIRECTIONAL_NAMES.index(value)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	650 elif k == 5:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	651 #print "DECOMP",hex(i), old.table[i][k], new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	652 # We assume that all normalization changes are in 1:1 mappings
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	653 assert " " not in value
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	654 normalization_changes.append((i, value))
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	655 elif k == 6:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	656 #print "DECIMAL",hex(i), old.table[i][k], new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	657 # we only support changes where the old value is a single digit
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	658 assert value in "0123456789"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	659 decimal_changes[i] = int(value)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	660 elif k == 8:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	661 # print "NUMERIC",hex(i), `old.table[i][k]`, new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	662 # Since 0 encodes "no change", the old value is better not 0
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	663 assert value != "0" and value != "-1"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	664 if not value:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	665 numeric_changes[i] = -1
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	666 else:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	667 assert re.match("^[0-9]+$", value)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	668 numeric_changes[i] = int(value)
49124 c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	669 elif k == 9:
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	670 if value == 'Y':
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	671 mirrored_changes[i] = '1'
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	672 else:
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	673 mirrored_changes[i] = '0'
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	674 elif k == 11:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	675 # change to ISO comment, ignore
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	676 pass
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	677 elif k == 12:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	678 # change to simple uppercase mapping; ignore
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	679 pass
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	680 elif k == 13:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	681 # change to simple lowercase mapping; ignore
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	682 pass
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	683 elif k == 14:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	684 # change to simple titlecase mapping; ignore
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	685 pass
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	686 else:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	687 class Difference(Exception):pass
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	688 raise Difference, (hex(i), k, old.table[i], new.table[i])
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	689 new.changed.append((version, zip(bidir_changes, category_changes,
49124 c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	690 decimal_changes, mirrored_changes,
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	691 numeric_changes),
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	692 normalization_changes))
36937 df43c04d2554 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 36924 diff changeset	693
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	694
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	695 # --------------------------------------------------------------------
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	696 # the following support code is taken from the unidb utilities
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	697 # Copyright (c) 1999-2000 by Secret Labs AB
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	698
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	699 # load a unicode-data file from disk
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	700
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	701 import sys
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	702
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	703 class UnicodeData:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	704
53553 019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	705 def __init__(self, filename, exclusions, eastasianwidth,
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	706 derivednormalizationprops=None, expand=1):
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	707 self.changed = []
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	708 file = open(filename)
25852 1f5f6481886d Update to Unicode 3.2 database. Martin v. Löwis <martin@v.loewis.de> parents: 25219 diff changeset	709 table = [None] * 0x110000
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	710 while 1:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	711 s = file.readline()
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	712 if not s:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	713 break
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	714 s = s.strip().split(";")
8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	715 char = int(s[0], 16)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	716 table[char] = s
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	717
26394 491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	718 # expand first-last ranges
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	719 if expand:
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	720 field = None
26394 491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	721 for i in range(0, 0x110000):
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	722 s = table[i]
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	723 if s:
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	724 if s[1][-6:] == "First>":
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	725 s[1] = ""
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	726 field = s
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	727 elif s[1][-5:] == "Last>":
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	728 s[1] = ""
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	729 field = None
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	730 elif field:
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	731 f2 = field[:]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	732 f2[0] = "%X" % i
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	733 table[i] = f2
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	734
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	735 # public attributes
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	736 self.filename = filename
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	737 self.table = table
25852 1f5f6481886d Update to Unicode 3.2 database. Martin v. Löwis <martin@v.loewis.de> parents: 25219 diff changeset	738 self.chars = range(0x110000) # unicode 3.2
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	739
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	740 file = open(exclusions)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	741 self.exclusions = {}
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	742 for s in file:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	743 s = s.strip()
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	744 if not s:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	745 continue
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	746 if s[0] == '#':
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	747 continue
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	748 char = int(s.split()[0],16)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	749 self.exclusions[char] = 1
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	750
32197 3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	751 widths = [None] * 0x110000
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	752 for s in open(eastasianwidth):
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	753 s = s.strip()
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	754 if not s:
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	755 continue
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	756 if s[0] == '#':
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	757 continue
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	758 s = s.split()[0].split(';')
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	759 if '..' in s[0]:
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	760 first, last = [int(c, 16) for c in s[0].split('..')]
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	761 chars = range(first, last+1)
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	762 else:
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	763 chars = [int(s[0], 16)]
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	764 for char in chars:
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	765 widths[char] = s[1]
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	766 for i in range(0, 0x110000):
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	767 if table[i] is not None:
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	768 table[i].append(widths[i])
53553 019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	769 if derivednormalizationprops:
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	770 quickchecks = [0] * 0x110000 # default is Yes
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	771 qc_order = 'NFD_QC NFKD_QC NFC_QC NFKC_QC'.split()
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	772 for s in open(derivednormalizationprops):
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	773 if '#' in s:
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	774 s = s[:s.index('#')]
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	775 s = [i.strip() for i in s.split(';')]
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	776 if len(s) < 2 or s[1] not in qc_order:
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	777 continue
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	778 quickcheck = 'MN'.index(s[2]) + 1 # Maybe or No
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	779 quickcheck_shift = qc_order.index(s[1])*2
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	780 quickcheck <<= quickcheck_shift
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	781 if '..' not in s[0]:
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	782 first = last = int(s[0], 16)
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	783 else:
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	784 first, last = [int(c, 16) for c in s[0].split('..')]
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	785 for char in range(first, last+1):
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	786 assert not (quickchecks[char]>>quickcheck_shift)&3
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	787 quickchecks[char] \|= quickcheck
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	788 for i in range(0, 0x110000):
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	789 if table[i] is not None:
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	790 table[i].append(quickchecks[i])
32197 3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	791
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	792 def uselatin1(self):
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	793 # restrict character range to ISO Latin 1
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	794 self.chars = range(256)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	795
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	796 # hash table tools
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	797
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	798 # this is a straight-forward reimplementation of Python's built-in
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	799 # dictionary type, using a static data structure, and a custom string
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	800 # hash algorithm.
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	801
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	802 def myhash(s, magic):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	803 h = 0
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	804 for c in map(ord, s.upper()):
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	805 h = (h * magic) + c
26394 491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	806 ix = h & 0xff000000L
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	807 if ix:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	808 h = (h ^ ((ix>>24) & 0xff)) & 0x00ffffff
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	809 return h
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	810
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	811 SIZES = [
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	812 (4,3), (8,3), (16,3), (32,5), (64,3), (128,3), (256,29), (512,17),
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	813 (1024,9), (2048,5), (4096,83), (8192,27), (16384,43), (32768,3),
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	814 (65536,45), (131072,9), (262144,39), (524288,39), (1048576,9),
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	815 (2097152,5), (4194304,3), (8388608,33), (16777216,27)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	816 ]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	817
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	818 class Hash:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	819 def __init__(self, name, data, magic):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	820 # turn a (key, value) list into a static hash table structure
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	821
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	822 # determine table size
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	823 for size, poly in SIZES:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	824 if size > len(data):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	825 poly = size + poly
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	826 break
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	827 else:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	828 raise AssertionError, "ran out of polynominals"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	829
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	830 print size, "slots in hash table"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	831
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	832 table = [None] * size
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	833
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	834 mask = size-1
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	835
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	836 n = 0
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	837
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	838 hash = myhash
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	839
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	840 # initialize hash table
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	841 for key, value in data:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	842 h = hash(key, magic)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	843 i = (~h) & mask
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	844 v = table[i]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	845 if v is None:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	846 table[i] = value
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	847 continue
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	848 incr = (h ^ (h >> 3)) & mask;
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	849 if not incr:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	850 incr = mask
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	851 while 1:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	852 n = n + 1
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	853 i = (i + incr) & mask
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	854 v = table[i]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	855 if v is None:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	856 table[i] = value
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	857 break
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	858 incr = incr << 1
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	859 if incr > mask:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	860 incr = incr ^ poly
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	861
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	862 print n, "collisions"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	863 self.collisions = n
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	864
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	865 for i in range(len(table)):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	866 if table[i] is None:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	867 table[i] = 0
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	868
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	869 self.data = Array(name + "_hash", table)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	870 self.magic = magic
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	871 self.name = name
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	872 self.size = size
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	873 self.poly = poly
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	874
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	875 def dump(self, file, trace):
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	876 # write data to file, as a C array
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	877 self.data.dump(file, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	878 file.write("#define %s_magic %d\n" % (self.name, self.magic))
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	879 file.write("#define %s_size %d\n" % (self.name, self.size))
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	880 file.write("#define %s_poly %d\n" % (self.name, self.poly))
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	881
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	882 # stuff to deal with arrays of unsigned integers
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	883
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	884 class Array:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	885
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	886 def __init__(self, name, data):
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	887 self.name = name
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	888 self.data = data
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	889
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	890 def dump(self, file, trace=0):
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	891 # write data to file, as a C array
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	892 size = getsize(self.data)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	893 if trace:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	894 print >>sys.stderr, self.name+":", size*len(self.data), "bytes"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	895 file.write("static ")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	896 if size == 1:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	897 file.write("unsigned char")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	898 elif size == 2:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	899 file.write("unsigned short")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	900 else:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	901 file.write("unsigned int")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	902 file.write(" " + self.name + "[] = {\n")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	903 if self.data:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	904 s = " "
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	905 for item in self.data:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	906 i = str(item) + ", "
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	907 if len(s) + len(i) > 78:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	908 file.write(s + "\n")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	909 s = " " + i
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	910 else:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	911 s = s + i
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	912 if s.strip():
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	913 file.write(s + "\n")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	914 file.write("};\n\n")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	915
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	916 def getsize(data):
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	917 # return smallest possible integer size for the given array
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	918 maxdata = max(data)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	919 if maxdata < 256:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	920 return 1
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	921 elif maxdata < 65536:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	922 return 2
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	923 else:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	924 return 4
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	925
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	926 def splitbins(t, trace=0):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	927 """t, trace=0 -> (t1, t2, shift). Split a table to save space.
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	928
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	929 t is a sequence of ints. This function can be useful to save space if
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	930 many of the ints are the same. t1 and t2 are lists of ints, and shift
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	931 is an int, chosen to minimize the combined size of t1 and t2 (in C
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	932 code), and where for each i in range(len(t)),
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	933 t[i] == t2[(t1[i >> shift] << shift) + (i & mask)]
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	934 where mask is a bitmask isolating the last "shift" bits.
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	935
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	936 If optional arg trace is non-zero (default zero), progress info
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	937 is printed to sys.stderr. The higher the value, the more info
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	938 you'll get.
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	939 """
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	940
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	941 import sys
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	942 if trace:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	943 def dump(t1, t2, shift, bytes):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	944 print >>sys.stderr, "%d+%d bins at shift %d; %d bytes" % (
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	945 len(t1), len(t2), shift, bytes)
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	946 print >>sys.stderr, "Size of original table:", len(t)*getsize(t), \
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	947 "bytes"
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	948 n = len(t)-1 # last valid index
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	949 maxshift = 0 # the most we can shift n and still have something left
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	950 if n > 0:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	951 while n >> 1:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	952 n >>= 1
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	953 maxshift += 1
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	954 del n
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	955 bytes = sys.maxint # smallest total size so far
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	956 t = tuple(t) # so slices can be dict keys
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	957 for shift in range(maxshift + 1):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	958 t1 = []
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	959 t2 = []
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	960 size = 2**shift
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	961 bincache = {}
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	962 for i in range(0, len(t), size):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	963 bin = t[i:i+size]
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	964 index = bincache.get(bin)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	965 if index is None:
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	966 index = len(t2)
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	967 bincache[bin] = index
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	968 t2.extend(bin)
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	969 t1.append(index >> shift)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	970 # determine memory size
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	971 b = len(t1)getsize(t1) + len(t2)getsize(t2)
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	972 if trace > 1:
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	973 dump(t1, t2, shift, b)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	974 if b < bytes:
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	975 best = t1, t2, shift
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	976 bytes = b
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	977 t1, t2, shift = best
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	978 if trace:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	979 print >>sys.stderr, "Best:",
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	980 dump(t1, t2, shift, bytes)
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	981 if __debug__:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	982 # exhaustively verify that the decomposition is correct
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	983 mask = ~((~0) << shift) # i.e., low-bit mask of shift bits
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	984 for i in xrange(len(t)):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	985 assert t[i] == t2[(t1[i >> shift] << shift) + (i & mask)]
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	986 return best
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	987
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	988 if __name__ == "__main__":
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	989 maketables(1)

Mercurial > cpython-lru_cache-weakref

annotate Tools/unicode/makeunicodedata.py @ 54294:dba767bc3155 legacy-trunk