cpython-lru_cache-weakref: Tools/unicode/makeunicodedata.py annotate

annotate Tools/unicode/makeunicodedata.py @ 58765:2ba7677bfa95 legacy-trunk

testCopysign was defined twice in test_math; combine the definitions

author	Mark Dickinson <dickinsm@gmail.com>
date	Sat, 06 Feb 2010 23:11:25 +0000
parents	7fe8a8313a5b
children	8aa5530e8407

rev	line source
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1 #
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	2 # (re)generate unicode property and type databases
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	3 #
26397 b27c1cad77ea Add unidata_version. Bump generator version number. Martin v. Löwis <martin@v.loewis.de> parents: 26394 diff changeset	4 # this script converts a unicode 3.2 database file to
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	5 # Modules/unicodedata_db.h, Modules/unicodename_db.h,
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	6 # and Objects/unicodetype_db.h
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	7 #
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	8 # history:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	9 # 2000-09-24 fl created (based on bits and pieces from unidb)
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	10 # 2000-09-25 fl merged tim's splitbin fixes, separate decomposition table
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	11 # 2000-09-25 fl added character type table
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	12 # 2000-09-26 fl added LINEBREAK, DECIMAL, and DIGIT flags/fields (2.0)
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	13 # 2000-11-03 fl expand first/last ranges
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	14 # 2001-01-19 fl added character name tables (2.1)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	15 # 2001-01-21 fl added decomp compression; dynamic phrasebook threshold
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	16 # 2002-09-11 wd use string methods
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	17 # 2002-10-18 mvl update to Unicode 3.2
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	18 # 2002-10-22 mvl generate NFC tables
26394 491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	19 # 2002-11-24 mvl expand all ranges, sort names version-independently
26397 b27c1cad77ea Add unidata_version. Bump generator version number. Martin v. Löwis <martin@v.loewis.de> parents: 26394 diff changeset	20 # 2002-11-25 mvl add UNIDATA_VERSION
32197 3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	21 # 2004-05-29 perky add east asian width information
36940 e4465d2737c4 Add changelog entry. Martin v. Löwis <martin@v.loewis.de> parents: 36937 diff changeset	22 # 2006-03-10 mvl update to Unicode 4.1; add UCD 3.2 delta
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	23 #
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	24 # written by Fredrik Lundh (fredrik@pythonware.com)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	25 #
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	26
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	27 import sys
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	28
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	29 SCRIPT = sys.argv[0]
49124 c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	30 VERSION = "2.6"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	31
26397 b27c1cad77ea Add unidata_version. Bump generator version number. Martin v. Löwis <martin@v.loewis.de> parents: 26394 diff changeset	32 # The Unicode Database
49124 c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	33 UNIDATA_VERSION = "5.1.0"
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	34 UNICODE_DATA = "UnicodeData%s.txt"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	35 COMPOSITION_EXCLUSIONS = "CompositionExclusions%s.txt"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	36 EASTASIAN_WIDTH = "EastAsianWidth%s.txt"
56406 a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	37 UNIHAN = "Unihan%s.txt"
53553 019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	38 DERIVEDNORMALIZATION_PROPS = "DerivedNormalizationProps%s.txt"
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	39
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	40 old_versions = ["3.2.0"]
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	41
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	42 CATEGORY_NAMES = [ "Cn", "Lu", "Ll", "Lt", "Mn", "Mc", "Me", "Nd",
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	43 "Nl", "No", "Zs", "Zl", "Zp", "Cc", "Cf", "Cs", "Co", "Cn", "Lm",
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	44 "Lo", "Pc", "Pd", "Ps", "Pe", "Pi", "Pf", "Po", "Sm", "Sc", "Sk",
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	45 "So" ]
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	46
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	47 BIDIRECTIONAL_NAMES = [ "", "L", "LRE", "LRO", "R", "AL", "RLE", "RLO",
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	48 "PDF", "EN", "ES", "ET", "AN", "CS", "NSM", "BN", "B", "S", "WS",
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	49 "ON" ]
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	50
32951 024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	51 EASTASIANWIDTH_NAMES = [ "F", "H", "W", "Na", "A", "N" ]
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	52
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	53 # note: should match definitions in Objects/unicodectype.c
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	54 ALPHA_MASK = 0x01
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	55 DECIMAL_MASK = 0x02
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	56 DIGIT_MASK = 0x04
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	57 LOWER_MASK = 0x08
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	58 LINEBREAK_MASK = 0x10
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	59 SPACE_MASK = 0x20
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	60 TITLE_MASK = 0x40
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	61 UPPER_MASK = 0x80
49124 c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	62 NODELTA_MASK = 0x100
56406 a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	63 NUMERIC_MASK = 0x200
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	64
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	65 def maketables(trace=0):
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	66
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	67 print "--- Reading", UNICODE_DATA % "", "..."
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	68
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	69 version = ""
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	70 unicode = UnicodeData(UNICODE_DATA % version,
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	71 COMPOSITION_EXCLUSIONS % version,
53553 019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	72 EASTASIAN_WIDTH % version,
56406 a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	73 UNIHAN % version,
53553 019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	74 DERIVEDNORMALIZATION_PROPS % version)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	75
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	76 print len(filter(None, unicode.table)), "characters"
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	77
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	78 for version in old_versions:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	79 print "--- Reading", UNICODE_DATA % ("-"+version), "..."
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	80 old_unicode = UnicodeData(UNICODE_DATA % ("-"+version),
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	81 COMPOSITION_EXCLUSIONS % ("-"+version),
56406 a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	82 EASTASIAN_WIDTH % ("-"+version),
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	83 UNIHAN % ("-"+version))
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	84 print len(filter(None, old_unicode.table)), "characters"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	85 merge_old_version(version, unicode, old_unicode)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	86
16091 45c065668ea5 Unicode nits: Don't include unicodedatabase.h no more. And make sure Fredrik Lundh <fredrik@pythonware.com> parents: 16090 diff changeset	87 makeunicodename(unicode, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	88 makeunicodedata(unicode, trace)
16091 45c065668ea5 Unicode nits: Don't include unicodedatabase.h no more. And make sure Fredrik Lundh <fredrik@pythonware.com> parents: 16090 diff changeset	89 makeunicodetype(unicode, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	90
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	91 # --------------------------------------------------------------------
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	92 # unicode character properties
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	93
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	94 def makeunicodedata(unicode, trace):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	95
53553 019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	96 dummy = (0, 0, 0, 0, 0, 0)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	97 table = [dummy]
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	98 cache = {0: dummy}
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	99 index = [0] * len(unicode.chars)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	100
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	101 FILE = "Modules/unicodedata_db.h"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	102
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	103 print "--- Preparing", FILE, "..."
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	104
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	105 # 1) database properties
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	106
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	107 for char in unicode.chars:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	108 record = unicode.table[char]
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	109 if record:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	110 # extract database properties
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	111 category = CATEGORY_NAMES.index(record[2])
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	112 combining = int(record[3])
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	113 bidirectional = BIDIRECTIONAL_NAMES.index(record[4])
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	114 mirrored = record[9] == "Y"
32951 024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	115 eastasianwidth = EASTASIANWIDTH_NAMES.index(record[15])
53553 019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	116 normalizationquickcheck = record[16]
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	117 item = (
53553 019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	118 category, combining, bidirectional, mirrored, eastasianwidth,
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	119 normalizationquickcheck
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	120 )
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	121 # add entry to index and item tables
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	122 i = cache.get(item)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	123 if i is None:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	124 cache[item] = i = len(table)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	125 table.append(item)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	126 index[char] = i
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	127
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	128 # 2) decomposition data
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	129
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	130 decomp_data = [0]
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	131 decomp_prefix = [""]
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	132 decomp_index = [0] * len(unicode.chars)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	133 decomp_size = 0
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	134
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	135 comp_pairs = []
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	136 comp_first = [None] * len(unicode.chars)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	137 comp_last = [None] * len(unicode.chars)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	138
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	139 for char in unicode.chars:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	140 record = unicode.table[char]
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	141 if record:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	142 if record[5]:
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	143 decomp = record[5].split()
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	144 if len(decomp) > 19:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	145 raise Exception, "character %x has a decomposition too large for nfd_nfkd" % char
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	146 # prefix
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	147 if decomp[0][0] == "<":
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	148 prefix = decomp.pop(0)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	149 else:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	150 prefix = ""
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	151 try:
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	152 i = decomp_prefix.index(prefix)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	153 except ValueError:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	154 i = len(decomp_prefix)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	155 decomp_prefix.append(prefix)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	156 prefix = i
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	157 assert prefix < 256
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	158 # content
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	159 decomp = [prefix + (len(decomp)<<8)] +\
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	160 map(lambda s: int(s, 16), decomp)
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	161 # Collect NFC pairs
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	162 if not prefix and len(decomp) == 3 and \
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	163 char not in unicode.exclusions and \
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	164 unicode.table[decomp[1]][3] == "0":
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	165 p, l, r = decomp
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	166 comp_first[l] = 1
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	167 comp_last[r] = 1
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	168 comp_pairs.append((l,r,char))
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	169 try:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	170 i = decomp_data.index(decomp)
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	171 except ValueError:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	172 i = len(decomp_data)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	173 decomp_data.extend(decomp)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	174 decomp_size = decomp_size + len(decomp) * 2
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	175 else:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	176 i = 0
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	177 decomp_index[char] = i
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	178
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	179 f = l = 0
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	180 comp_first_ranges = []
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	181 comp_last_ranges = []
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	182 prev_f = prev_l = None
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	183 for i in unicode.chars:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	184 if comp_first[i] is not None:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	185 comp_first[i] = f
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	186 f += 1
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	187 if prev_f is None:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	188 prev_f = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	189 elif prev_f[1]+1 == i:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	190 prev_f = prev_f[0],i
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	191 else:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	192 comp_first_ranges.append(prev_f)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	193 prev_f = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	194 if comp_last[i] is not None:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	195 comp_last[i] = l
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	196 l += 1
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	197 if prev_l is None:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	198 prev_l = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	199 elif prev_l[1]+1 == i:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	200 prev_l = prev_l[0],i
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	201 else:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	202 comp_last_ranges.append(prev_l)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	203 prev_l = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	204 comp_first_ranges.append(prev_f)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	205 comp_last_ranges.append(prev_l)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	206 total_first = f
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	207 total_last = l
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	208
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	209 comp_data = [0](total_firsttotal_last)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	210 for f,l,char in comp_pairs:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	211 f = comp_first[f]
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	212 l = comp_last[l]
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	213 comp_data[f*total_last+l] = char
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	214
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	215 print len(table), "unique properties"
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	216 print len(decomp_prefix), "unique decomposition prefixes"
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	217 print len(decomp_data), "unique decomposition entries:",
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	218 print decomp_size, "bytes"
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	219 print total_first, "first characters in NFC"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	220 print total_last, "last characters in NFC"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	221 print len(comp_pairs), "NFC pairs"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	222
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	223 print "--- Writing", FILE, "..."
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	224
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	225 fp = open(FILE, "w")
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	226 print >>fp, "/* this file was generated by %s %s */" % (SCRIPT, VERSION)
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	227 print >>fp
26397 b27c1cad77ea Add unidata_version. Bump generator version number. Martin v. Löwis <martin@v.loewis.de> parents: 26394 diff changeset	228 print >>fp, '#define UNIDATA_VERSION "%s"' % UNIDATA_VERSION
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	229 print >>fp, "/* a list of unique database records */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	230 print >>fp, \
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	231 "const _PyUnicode_DatabaseRecord _PyUnicode_Database_Records[] = {"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	232 for item in table:
53553 019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	233 print >>fp, " {%d, %d, %d, %d, %d, %d}," % item
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	234 print >>fp, "};"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	235 print >>fp
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	236
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	237 print >>fp, "/* Reindexing of NFC first characters. */"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	238 print >>fp, "#define TOTAL_FIRST",total_first
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	239 print >>fp, "#define TOTAL_LAST",total_last
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	240 print >>fp, "struct reindex{int start;short count,index;};"
47678 a35da94e05fc Make more symbols static. Martin v. Löwis <martin@v.loewis.de> parents: 36940 diff changeset	241 print >>fp, "static struct reindex nfc_first[] = {"
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	242 for start,end in comp_first_ranges:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	243 print >>fp," { %d, %d, %d}," % (start,end-start,comp_first[start])
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	244 print >>fp," {0,0,0}"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	245 print >>fp,"};\n"
47678 a35da94e05fc Make more symbols static. Martin v. Löwis <martin@v.loewis.de> parents: 36940 diff changeset	246 print >>fp, "static struct reindex nfc_last[] = {"
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	247 for start,end in comp_last_ranges:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	248 print >>fp," { %d, %d, %d}," % (start,end-start,comp_last[start])
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	249 print >>fp," {0,0,0}"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	250 print >>fp,"};\n"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	251
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	252 # FIXME: <fl> the following tables could be made static, and
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	253 # the support code moved into unicodedatabase.c
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	254
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	255 print >>fp, "/* string literals */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	256 print >>fp, "const char *_PyUnicode_CategoryNames[] = {"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	257 for name in CATEGORY_NAMES:
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	258 print >>fp, " \"%s\"," % name
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	259 print >>fp, " NULL"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	260 print >>fp, "};"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	261
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	262 print >>fp, "const char *_PyUnicode_BidirectionalNames[] = {"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	263 for name in BIDIRECTIONAL_NAMES:
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	264 print >>fp, " \"%s\"," % name
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	265 print >>fp, " NULL"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	266 print >>fp, "};"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	267
32951 024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	268 print >>fp, "const char *_PyUnicode_EastAsianWidthNames[] = {"
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	269 for name in EASTASIANWIDTH_NAMES:
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	270 print >>fp, " \"%s\"," % name
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	271 print >>fp, " NULL"
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	272 print >>fp, "};"
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	273
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	274 print >>fp, "static const char *decomp_prefix[] = {"
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	275 for name in decomp_prefix:
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	276 print >>fp, " \"%s\"," % name
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	277 print >>fp, " NULL"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	278 print >>fp, "};"
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	279
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	280 # split record index table
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	281 index1, index2, shift = splitbins(index, trace)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	282
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	283 print >>fp, "/* index tables for the database records */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	284 print >>fp, "#define SHIFT", shift
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	285 Array("index1", index1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	286 Array("index2", index2).dump(fp, trace)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	287
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	288 # split decomposition index table
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	289 index1, index2, shift = splitbins(decomp_index, trace)
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	290
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	291 print >>fp, "/* decomposition data */"
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	292 Array("decomp_data", decomp_data).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	293
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	294 print >>fp, "/* index tables for the decomposition data */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	295 print >>fp, "#define DECOMP_SHIFT", shift
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	296 Array("decomp_index1", index1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	297 Array("decomp_index2", index2).dump(fp, trace)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	298
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	299 index, index2, shift = splitbins(comp_data, trace)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	300 print >>fp, "/* NFC pairs */"
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	301 print >>fp, "#define COMP_SHIFT", shift
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	302 Array("comp_index", index).dump(fp, trace)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	303 Array("comp_data", index2).dump(fp, trace)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	304
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	305 # Generate delta tables for old versions
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	306 for version, table, normalization in unicode.changed:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	307 cversion = version.replace(".","_")
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	308 records = [table[0]]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	309 cache = {table[0]:0}
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	310 index = [0] * len(table)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	311 for i, record in enumerate(table):
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	312 try:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	313 index[i] = cache[record]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	314 except KeyError:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	315 index[i] = cache[record] = len(records)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	316 records.append(record)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	317 index1, index2, shift = splitbins(index, trace)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	318 print >>fp, "static const change_record change_records_%s[] = {" % cversion
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	319 for record in records:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	320 print >>fp, "\t{ %s }," % ", ".join(map(str,record))
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	321 print >>fp, "};"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	322 Array("changes_%s_index" % cversion, index1).dump(fp, trace)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	323 Array("changes_%s_data" % cversion, index2).dump(fp, trace)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	324 print >>fp, "static const change_record* get_change_%s(Py_UCS4 n)" % cversion
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	325 print >>fp, "{"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	326 print >>fp, "\tint index;"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	327 print >>fp, "\tif (n >= 0x110000) index = 0;"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	328 print >>fp, "\telse {"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	329 print >>fp, "\t\tindex = changes_%s_index[n>>%d];" % (cversion, shift)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	330 print >>fp, "\t\tindex = changes_%s_data[(index<<%d)+(n & %d)];" % \
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	331 (cversion, shift, ((1<<shift)-1))
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	332 print >>fp, "\t}"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	333 print >>fp, "\treturn change_records_%s+index;" % cversion
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	334 print >>fp, "}\n"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	335 print >>fp, "static Py_UCS4 normalization_%s(Py_UCS4 n)" % cversion
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	336 print >>fp, "{"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	337 print >>fp, "\tswitch(n) {"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	338 for k, v in normalization:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	339 print >>fp, "\tcase %s: return 0x%s;" % (hex(k), v)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	340 print >>fp, "\tdefault: return 0;"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	341 print >>fp, "\t}\n}\n"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	342
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	343 fp.close()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	344
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	345 # --------------------------------------------------------------------
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	346 # unicode character type tables
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	347
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	348 def makeunicodetype(unicode, trace):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	349
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	350 FILE = "Objects/unicodetype_db.h"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	351
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	352 print "--- Preparing", FILE, "..."
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	353
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	354 # extract unicode types
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	355 dummy = (0, 0, 0, 0, 0, 0)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	356 table = [dummy]
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	357 cache = {0: dummy}
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	358 index = [0] * len(unicode.chars)
56406 a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	359 numeric = {}
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	360 spaces = []
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	361 linebreaks = []
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	362
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	363 for char in unicode.chars:
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	364 record = unicode.table[char]
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	365 if record:
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	366 # extract database properties
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	367 category = record[2]
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	368 bidirectional = record[4]
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	369 flags = 0
49124 c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	370 delta = True
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	371 if category in ["Lm", "Lt", "Lu", "Ll", "Lo"]:
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	372 flags \|= ALPHA_MASK
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	373 if category == "Ll":
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	374 flags \|= LOWER_MASK
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	375 if category == "Zl" or bidirectional == "B":
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	376 flags \|= LINEBREAK_MASK
56406 a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	377 linebreaks.append(char)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	378 if category == "Zs" or bidirectional in ("WS", "B", "S"):
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	379 flags \|= SPACE_MASK
56406 a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	380 spaces.append(char)
14856 83aff1ea5b70 - don't set the titlecase flag for uppercase letters (sorry, tim) Fredrik Lundh <fredrik@pythonware.com> parents: 14852 diff changeset	381 if category == "Lt":
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	382 flags \|= TITLE_MASK
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	383 if category == "Lu":
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	384 flags \|= UPPER_MASK
49124 c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	385 # use delta predictor for upper/lower/title if it fits
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	386 if record[12]:
53400 f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	387 upper = int(record[12], 16)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	388 else:
53400 f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	389 upper = char
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	390 if record[13]:
53400 f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	391 lower = int(record[13], 16)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	392 else:
53400 f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	393 lower = char
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	394 if record[14]:
53400 f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	395 title = int(record[14], 16)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	396 else:
53400 f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	397 # UCD.html says that a missing title char means that
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	398 # it defaults to the uppercase character, not to the
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	399 # character itself. Apparently, in the current UCD (5.x)
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	400 # this feature is never used
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	401 title = upper
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	402 upper_d = upper - char
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	403 lower_d = lower - char
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	404 title_d = title - char
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	405 if -32768 <= upper_d <= 32767 and \
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	406 -32768 <= lower_d <= 32767 and \
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	407 -32768 <= title_d <= 32767:
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	408 # use deltas
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	409 upper = upper_d & 0xffff
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	410 lower = lower_d & 0xffff
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	411 title = title_d & 0xffff
f08f776082ec Merged revisions 71894 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 49124 diff changeset	412 else:
49124 c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	413 flags \|= NODELTA_MASK
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	414 # decimal digit, integer digit
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	415 decimal = 0
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	416 if record[6]:
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	417 flags \|= DECIMAL_MASK
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	418 decimal = int(record[6])
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	419 digit = 0
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	420 if record[7]:
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	421 flags \|= DIGIT_MASK
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	422 digit = int(record[7])
56406 a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	423 if record[8]:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	424 flags \|= NUMERIC_MASK
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	425 numeric.setdefault(record[8], []).append(char)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	426 item = (
32197 3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	427 upper, lower, title, decimal, digit, flags
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	428 )
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	429 # add entry to index and item tables
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	430 i = cache.get(item)
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	431 if i is None:
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	432 cache[item] = i = len(table)
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	433 table.append(item)
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	434 index[char] = i
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	435
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	436 print len(table), "unique character type entries"
56406 a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	437 print sum(map(len, numeric.values())), "numeric code points"
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	438 print len(spaces), "whitespace code points"
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	439 print len(linebreaks), "linebreak code points"
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	440
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	441 print "--- Writing", FILE, "..."
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	442
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	443 fp = open(FILE, "w")
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	444 print >>fp, "/* this file was generated by %s %s */" % (SCRIPT, VERSION)
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	445 print >>fp
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	446 print >>fp, "/* a list of unique character type descriptors */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	447 print >>fp, "const _PyUnicode_TypeRecord _PyUnicode_TypeRecords[] = {"
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	448 for item in table:
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	449 print >>fp, " {%d, %d, %d, %d, %d, %d}," % item
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	450 print >>fp, "};"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	451 print >>fp
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	452
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	453 # split decomposition index table
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	454 index1, index2, shift = splitbins(index, trace)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	455
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	456 print >>fp, "/* type indexes */"
c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	457 print >>fp, "#define SHIFT", shift
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	458 Array("index1", index1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	459 Array("index2", index2).dump(fp, trace)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	460
56406 a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	461 # Generate code for _PyUnicode_ToNumeric()
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	462 numeric_items = numeric.items()
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	463 numeric_items.sort()
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	464 print >>fp, '/* Returns the numeric value as double for Unicode characters'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	465 print >>fp, ' * having this property, -1.0 otherwise.'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	466 print >>fp, ' */'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	467 print >>fp, 'double _PyUnicode_ToNumeric(Py_UNICODE ch)'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	468 print >>fp, '{'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	469 print >>fp, ' switch (ch) {'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	470 for value, codepoints in numeric_items:
56507 7fe8a8313a5b #7112: Fix compilation warning in unicodetype_db.h Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 56406 diff changeset	471 # Turn text into float literals
7fe8a8313a5b #7112: Fix compilation warning in unicodetype_db.h Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 56406 diff changeset	472 parts = value.split('/')
7fe8a8313a5b #7112: Fix compilation warning in unicodetype_db.h Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 56406 diff changeset	473 parts = [repr(float(part)) for part in parts]
7fe8a8313a5b #7112: Fix compilation warning in unicodetype_db.h Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 56406 diff changeset	474 value = '/'.join(parts)
7fe8a8313a5b #7112: Fix compilation warning in unicodetype_db.h Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 56406 diff changeset	475
56406 a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	476 haswide = False
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	477 hasnonewide = False
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	478 codepoints.sort()
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	479 for codepoint in codepoints:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	480 if codepoint < 0x10000:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	481 hasnonewide = True
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	482 if codepoint >= 0x10000 and not haswide:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	483 print >>fp, '#ifdef Py_UNICODE_WIDE'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	484 haswide = True
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	485 print >>fp, ' case 0x%04X:' % (codepoint,)
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	486 if haswide and hasnonewide:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	487 print >>fp, '#endif'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	488 print >>fp, ' return (double) %s;' % (value,)
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	489 if haswide and not hasnonewide:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	490 print >>fp, '#endif'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	491 print >>fp,' }'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	492 print >>fp,' return -1.0;'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	493 print >>fp,'}'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	494 print >>fp
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	495
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	496 # Generate code for _PyUnicode_IsWhitespace()
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	497 print >>fp, "/* Returns 1 for Unicode characters having the bidirectional"
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	498 print >>fp, " * type 'WS', 'B' or 'S' or the category 'Zs', 0 otherwise."
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	499 print >>fp, " */"
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	500 print >>fp, 'int _PyUnicode_IsWhitespace(register const Py_UNICODE ch)'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	501 print >>fp, '{'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	502 print >>fp, '#ifdef WANT_WCTYPE_FUNCTIONS'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	503 print >>fp, ' return iswspace(ch);'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	504 print >>fp, '#else'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	505 print >>fp, ' switch (ch) {'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	506
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	507 haswide = False
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	508 hasnonewide = False
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	509 spaces.sort()
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	510 for codepoint in spaces:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	511 if codepoint < 0x10000:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	512 hasnonewide = True
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	513 if codepoint >= 0x10000 and not haswide:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	514 print >>fp, '#ifdef Py_UNICODE_WIDE'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	515 haswide = True
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	516 print >>fp, ' case 0x%04X:' % (codepoint,)
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	517 if haswide and hasnonewide:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	518 print >>fp, '#endif'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	519 print >>fp, ' return 1;'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	520 if haswide and not hasnonewide:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	521 print >>fp, '#endif'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	522
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	523 print >>fp,' }'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	524 print >>fp,' return 0;'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	525 print >>fp, '#endif'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	526 print >>fp,'}'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	527 print >>fp
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	528
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	529 # Generate code for _PyUnicode_IsLinebreak()
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	530 print >>fp, "/* Returns 1 for Unicode characters having the category 'Zl',"
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	531 print >>fp, " * 'Zp' or type 'B', 0 otherwise."
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	532 print >>fp, " */"
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	533 print >>fp, 'int _PyUnicode_IsLinebreak(register const Py_UNICODE ch)'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	534 print >>fp, '{'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	535 print >>fp, ' switch (ch) {'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	536 haswide = False
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	537 hasnonewide = False
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	538 linebreaks.sort()
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	539 for codepoint in linebreaks:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	540 if codepoint < 0x10000:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	541 hasnonewide = True
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	542 if codepoint >= 0x10000 and not haswide:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	543 print >>fp, '#ifdef Py_UNICODE_WIDE'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	544 haswide = True
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	545 print >>fp, ' case 0x%04X:' % (codepoint,)
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	546 if haswide and hasnonewide:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	547 print >>fp, '#endif'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	548 print >>fp, ' return 1;'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	549 if haswide and not hasnonewide:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	550 print >>fp, '#endif'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	551
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	552 print >>fp,' }'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	553 print >>fp,' return 0;'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	554 print >>fp,'}'
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	555 print >>fp
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	556
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	557 fp.close()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	558
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	559 # --------------------------------------------------------------------
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	560 # unicode name database
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	561
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	562 def makeunicodename(unicode, trace):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	563
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	564 FILE = "Modules/unicodename_db.h"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	565
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	566 print "--- Preparing", FILE, "..."
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	567
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	568 # collect names
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	569 names = [None] * len(unicode.chars)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	570
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	571 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	572 record = unicode.table[char]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	573 if record:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	574 name = record[1].strip()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	575 if name and name[0] != "<":
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	576 names[char] = name + chr(0)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	577
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	578 print len(filter(lambda n: n is not None, names)), "distinct names"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	579
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	580 # collect unique words from names (note that we differ between
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	581 # words inside a sentence, and words ending a sentence. the
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	582 # latter includes the trailing null byte.
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	583
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	584 words = {}
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	585 n = b = 0
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	586 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	587 name = names[char]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	588 if name:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	589 w = name.split()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	590 b = b + len(name)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	591 n = n + len(w)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	592 for w in w:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	593 l = words.get(w)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	594 if l:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	595 l.append(None)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	596 else:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	597 words[w] = [len(words)]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	598
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	599 print n, "words in text;", b, "bytes"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	600
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	601 wordlist = words.items()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	602
26394 491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	603 # sort on falling frequency, then by name
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	604 def cmpwords((aword, alist),(bword, blist)):
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	605 r = -cmp(len(alist),len(blist))
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	606 if r:
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	607 return r
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	608 return cmp(aword, bword)
491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	609 wordlist.sort(cmpwords)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	610
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	611 # figure out how many phrasebook escapes we need
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	612 escapes = 0
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	613 while escapes * 256 < len(wordlist):
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	614 escapes = escapes + 1
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	615 print escapes, "escapes"
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	616
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	617 short = 256 - escapes
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	618
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	619 assert short > 0
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	620
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	621 print short, "short indexes in lexicon"
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	622
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	623 # statistics
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	624 n = 0
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	625 for i in range(short):
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	626 n = n + len(wordlist[i][1])
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	627 print n, "short indexes in phrasebook"
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	628
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	629 # pick the most commonly used words, and sort the rest on falling
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	630 # length (to maximize overlap)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	631
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	632 wordlist, wordtail = wordlist[:short], wordlist[short:]
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	633 wordtail.sort(lambda a, b: len(b[0])-len(a[0]))
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	634 wordlist.extend(wordtail)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	635
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	636 # generate lexicon from words
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	637
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	638 lexicon_offset = [0]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	639 lexicon = ""
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	640 words = {}
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	641
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	642 # build a lexicon string
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	643 offset = 0
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	644 for w, x in wordlist:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	645 # encoding: bit 7 indicates last character in word (chr(128)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	646 # indicates the last character in an entire string)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	647 ww = w[:-1] + chr(ord(w[-1])+128)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	648 # reuse string tails, when possible
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	649 o = lexicon.find(ww)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	650 if o < 0:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	651 o = offset
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	652 lexicon = lexicon + ww
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	653 offset = offset + len(w)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	654 words[w] = len(lexicon_offset)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	655 lexicon_offset.append(o)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	656
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	657 lexicon = map(ord, lexicon)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	658
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	659 # generate phrasebook from names and lexicon
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	660 phrasebook = [0]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	661 phrasebook_offset = [0] * len(unicode.chars)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	662 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	663 name = names[char]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	664 if name:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	665 w = name.split()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	666 phrasebook_offset[char] = len(phrasebook)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	667 for w in w:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	668 i = words[w]
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	669 if i < short:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	670 phrasebook.append(i)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	671 else:
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	672 # store as two bytes
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	673 phrasebook.append((i>>8) + short)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	674 phrasebook.append(i&255)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	675
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	676 assert getsize(phrasebook) == 1
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	677
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	678 #
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	679 # unicode name hash table
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	680
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	681 # extract names
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	682 data = []
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	683 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	684 record = unicode.table[char]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	685 if record:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	686 name = record[1].strip()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	687 if name and name[0] != "<":
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	688 data.append((name, char))
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	689
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	690 # the magic number 47 was chosen to minimize the number of
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	691 # collisions on the current data set. if you like, change it
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	692 # and see what happens...
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	693
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	694 codehash = Hash("code", data, 47)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	695
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	696 print "--- Writing", FILE, "..."
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	697
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	698 fp = open(FILE, "w")
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	699 print >>fp, "/* this file was generated by %s %s */" % (SCRIPT, VERSION)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	700 print >>fp
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	701 print >>fp, "#define NAME_MAXLEN", 256
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	702 print >>fp
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	703 print >>fp, "/* lexicon */"
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	704 Array("lexicon", lexicon).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	705 Array("lexicon_offset", lexicon_offset).dump(fp, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	706
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	707 # split decomposition index table
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	708 offset1, offset2, shift = splitbins(phrasebook_offset, trace)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	709
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	710 print >>fp, "/* code->name phrasebook */"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	711 print >>fp, "#define phrasebook_shift", shift
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	712 print >>fp, "#define phrasebook_short", short
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	713
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	714 Array("phrasebook", phrasebook).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	715 Array("phrasebook_offset1", offset1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	716 Array("phrasebook_offset2", offset2).dump(fp, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	717
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	718 print >>fp, "/* name->code dictionary */"
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	719 codehash.dump(fp, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	720
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	721 fp.close()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	722
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	723
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	724 def merge_old_version(version, new, old):
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	725 # Changes to exclusion file not implemented yet
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	726 if old.exclusions != new.exclusions:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	727 raise NotImplementedError, "exclusions differ"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	728
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	729 # In these change records, 0xFF means "no change"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	730 bidir_changes = [0xFF]*0x110000
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	731 category_changes = [0xFF]*0x110000
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	732 decimal_changes = [0xFF]*0x110000
49124 c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	733 mirrored_changes = [0xFF]*0x110000
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	734 # In numeric data, 0 means "no change",
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	735 # -1 means "did not have a numeric value
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	736 numeric_changes = [0] * 0x110000
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	737 # normalization_changes is a list of key-value pairs
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	738 normalization_changes = []
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	739 for i in range(0x110000):
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	740 if new.table[i] is None:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	741 # Characters unassigned in the new version ought to
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	742 # be unassigned in the old one
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	743 assert old.table[i] is None
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	744 continue
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	745 # check characters unassigned in the old version
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	746 if old.table[i] is None:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	747 # category 0 is "unassigned"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	748 category_changes[i] = 0
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	749 continue
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	750 # check characters that differ
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	751 if old.table[i] != new.table[i]:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	752 for k in range(len(old.table[i])):
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	753 if old.table[i][k] != new.table[i][k]:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	754 value = old.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	755 if k == 2:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	756 #print "CATEGORY",hex(i), old.table[i][k], new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	757 category_changes[i] = CATEGORY_NAMES.index(value)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	758 elif k == 4:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	759 #print "BIDIR",hex(i), old.table[i][k], new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	760 bidir_changes[i] = BIDIRECTIONAL_NAMES.index(value)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	761 elif k == 5:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	762 #print "DECOMP",hex(i), old.table[i][k], new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	763 # We assume that all normalization changes are in 1:1 mappings
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	764 assert " " not in value
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	765 normalization_changes.append((i, value))
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	766 elif k == 6:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	767 #print "DECIMAL",hex(i), old.table[i][k], new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	768 # we only support changes where the old value is a single digit
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	769 assert value in "0123456789"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	770 decimal_changes[i] = int(value)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	771 elif k == 8:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	772 # print "NUMERIC",hex(i), `old.table[i][k]`, new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	773 # Since 0 encodes "no change", the old value is better not 0
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	774 if not value:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	775 numeric_changes[i] = -1
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	776 else:
56406 a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	777 numeric_changes[i] = float(value)
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	778 assert numeric_changes[i] not in (0, -1)
49124 c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	779 elif k == 9:
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	780 if value == 'Y':
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	781 mirrored_changes[i] = '1'
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	782 else:
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	783 mirrored_changes[i] = '0'
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	784 elif k == 11:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	785 # change to ISO comment, ignore
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	786 pass
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	787 elif k == 12:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	788 # change to simple uppercase mapping; ignore
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	789 pass
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	790 elif k == 13:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	791 # change to simple lowercase mapping; ignore
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	792 pass
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	793 elif k == 14:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	794 # change to simple titlecase mapping; ignore
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	795 pass
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	796 else:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	797 class Difference(Exception):pass
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	798 raise Difference, (hex(i), k, old.table[i], new.table[i])
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	799 new.changed.append((version, zip(bidir_changes, category_changes,
49124 c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	800 decimal_changes, mirrored_changes,
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47678 diff changeset	801 numeric_changes),
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	802 normalization_changes))
36937 df43c04d2554 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 36924 diff changeset	803
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	804
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	805 # --------------------------------------------------------------------
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	806 # the following support code is taken from the unidb utilities
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	807 # Copyright (c) 1999-2000 by Secret Labs AB
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	808
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	809 # load a unicode-data file from disk
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	810
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	811 class UnicodeData:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	812
56406 a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	813 def __init__(self, filename, exclusions, eastasianwidth, unihan,
53553 019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	814 derivednormalizationprops=None, expand=1):
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	815 self.changed = []
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	816 file = open(filename)
25852 1f5f6481886d Update to Unicode 3.2 database. Martin v. Löwis <martin@v.loewis.de> parents: 25219 diff changeset	817 table = [None] * 0x110000
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	818 while 1:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	819 s = file.readline()
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	820 if not s:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	821 break
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	822 s = s.strip().split(";")
8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	823 char = int(s[0], 16)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	824 table[char] = s
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	825
26394 491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	826 # expand first-last ranges
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	827 if expand:
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	828 field = None
26394 491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	829 for i in range(0, 0x110000):
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	830 s = table[i]
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	831 if s:
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	832 if s[1][-6:] == "First>":
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	833 s[1] = ""
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	834 field = s
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	835 elif s[1][-5:] == "Last>":
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	836 s[1] = ""
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	837 field = None
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	838 elif field:
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	839 f2 = field[:]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	840 f2[0] = "%X" % i
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	841 table[i] = f2
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	842
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	843 # public attributes
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	844 self.filename = filename
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	845 self.table = table
25852 1f5f6481886d Update to Unicode 3.2 database. Martin v. Löwis <martin@v.loewis.de> parents: 25219 diff changeset	846 self.chars = range(0x110000) # unicode 3.2
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	847
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	848 file = open(exclusions)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	849 self.exclusions = {}
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	850 for s in file:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	851 s = s.strip()
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	852 if not s:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	853 continue
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	854 if s[0] == '#':
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	855 continue
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	856 char = int(s.split()[0],16)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	857 self.exclusions[char] = 1
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	858
32197 3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	859 widths = [None] * 0x110000
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	860 for s in open(eastasianwidth):
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	861 s = s.strip()
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	862 if not s:
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	863 continue
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	864 if s[0] == '#':
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	865 continue
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	866 s = s.split()[0].split(';')
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	867 if '..' in s[0]:
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	868 first, last = [int(c, 16) for c in s[0].split('..')]
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	869 chars = range(first, last+1)
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	870 else:
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	871 chars = [int(s[0], 16)]
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	872 for char in chars:
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	873 widths[char] = s[1]
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	874 for i in range(0, 0x110000):
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	875 if table[i] is not None:
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	876 table[i].append(widths[i])
53553 019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	877 if derivednormalizationprops:
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	878 quickchecks = [0] * 0x110000 # default is Yes
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	879 qc_order = 'NFD_QC NFKD_QC NFC_QC NFKC_QC'.split()
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	880 for s in open(derivednormalizationprops):
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	881 if '#' in s:
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	882 s = s[:s.index('#')]
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	883 s = [i.strip() for i in s.split(';')]
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	884 if len(s) < 2 or s[1] not in qc_order:
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	885 continue
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	886 quickcheck = 'MN'.index(s[2]) + 1 # Maybe or No
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	887 quickcheck_shift = qc_order.index(s[1])*2
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	888 quickcheck <<= quickcheck_shift
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	889 if '..' not in s[0]:
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	890 first = last = int(s[0], 16)
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	891 else:
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	892 first, last = [int(c, 16) for c in s[0].split('..')]
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	893 for char in range(first, last+1):
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	894 assert not (quickchecks[char]>>quickcheck_shift)&3
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	895 quickchecks[char] \|= quickcheck
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	896 for i in range(0, 0x110000):
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	897 if table[i] is not None:
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 53400 diff changeset	898 table[i].append(quickchecks[i])
32197 3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	899
56406 a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	900 for line in open(unihan):
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	901 if not line.startswith('U+'):
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	902 continue
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	903 code, tag, value = line.split(None, 3)[:3]
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	904 if tag not in ('kAccountingNumeric', 'kPrimaryNumeric',
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	905 'kOtherNumeric'):
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	906 continue
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	907 value = value.strip().replace(',', '')
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	908 i = int(code[2:], 16)
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	909 # Patch the numeric field
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	910 if table[i] is not None:
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	911 table[i][8] = value
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	912
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	913 def uselatin1(self):
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	914 # restrict character range to ISO Latin 1
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	915 self.chars = range(256)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	916
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	917 # hash table tools
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	918
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	919 # this is a straight-forward reimplementation of Python's built-in
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	920 # dictionary type, using a static data structure, and a custom string
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	921 # hash algorithm.
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	922
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	923 def myhash(s, magic):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	924 h = 0
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	925 for c in map(ord, s.upper()):
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	926 h = (h * magic) + c
26394 491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	927 ix = h & 0xff000000L
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	928 if ix:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	929 h = (h ^ ((ix>>24) & 0xff)) & 0x00ffffff
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	930 return h
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	931
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	932 SIZES = [
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	933 (4,3), (8,3), (16,3), (32,5), (64,3), (128,3), (256,29), (512,17),
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	934 (1024,9), (2048,5), (4096,83), (8192,27), (16384,43), (32768,3),
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	935 (65536,45), (131072,9), (262144,39), (524288,39), (1048576,9),
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	936 (2097152,5), (4194304,3), (8388608,33), (16777216,27)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	937 ]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	938
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	939 class Hash:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	940 def __init__(self, name, data, magic):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	941 # turn a (key, value) list into a static hash table structure
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	942
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	943 # determine table size
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	944 for size, poly in SIZES:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	945 if size > len(data):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	946 poly = size + poly
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	947 break
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	948 else:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	949 raise AssertionError, "ran out of polynominals"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	950
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	951 print size, "slots in hash table"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	952
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	953 table = [None] * size
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	954
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	955 mask = size-1
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	956
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	957 n = 0
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	958
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	959 hash = myhash
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	960
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	961 # initialize hash table
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	962 for key, value in data:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	963 h = hash(key, magic)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	964 i = (~h) & mask
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	965 v = table[i]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	966 if v is None:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	967 table[i] = value
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	968 continue
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	969 incr = (h ^ (h >> 3)) & mask;
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	970 if not incr:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	971 incr = mask
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	972 while 1:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	973 n = n + 1
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	974 i = (i + incr) & mask
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	975 v = table[i]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	976 if v is None:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	977 table[i] = value
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	978 break
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	979 incr = incr << 1
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	980 if incr > mask:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	981 incr = incr ^ poly
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	982
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	983 print n, "collisions"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	984 self.collisions = n
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	985
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	986 for i in range(len(table)):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	987 if table[i] is None:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	988 table[i] = 0
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	989
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	990 self.data = Array(name + "_hash", table)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	991 self.magic = magic
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	992 self.name = name
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	993 self.size = size
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	994 self.poly = poly
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	995
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	996 def dump(self, file, trace):
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	997 # write data to file, as a C array
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	998 self.data.dump(file, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	999 file.write("#define %s_magic %d\n" % (self.name, self.magic))
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1000 file.write("#define %s_size %d\n" % (self.name, self.size))
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1001 file.write("#define %s_poly %d\n" % (self.name, self.poly))
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1002
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1003 # stuff to deal with arrays of unsigned integers
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1004
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1005 class Array:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1006
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1007 def __init__(self, name, data):
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1008 self.name = name
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1009 self.data = data
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1010
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	1011 def dump(self, file, trace=0):
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1012 # write data to file, as a C array
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1013 size = getsize(self.data)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	1014 if trace:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	1015 print >>sys.stderr, self.name+":", size*len(self.data), "bytes"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1016 file.write("static ")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1017 if size == 1:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1018 file.write("unsigned char")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1019 elif size == 2:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1020 file.write("unsigned short")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1021 else:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1022 file.write("unsigned int")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1023 file.write(" " + self.name + "[] = {\n")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1024 if self.data:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1025 s = " "
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1026 for item in self.data:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1027 i = str(item) + ", "
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1028 if len(s) + len(i) > 78:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1029 file.write(s + "\n")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1030 s = " " + i
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1031 else:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1032 s = s + i
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	1033 if s.strip():
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1034 file.write(s + "\n")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1035 file.write("};\n\n")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1036
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1037 def getsize(data):
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1038 # return smallest possible integer size for the given array
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1039 maxdata = max(data)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1040 if maxdata < 256:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1041 return 1
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1042 elif maxdata < 65536:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1043 return 2
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1044 else:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1045 return 4
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1046
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1047 def splitbins(t, trace=0):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1048 """t, trace=0 -> (t1, t2, shift). Split a table to save space.
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1049
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1050 t is a sequence of ints. This function can be useful to save space if
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1051 many of the ints are the same. t1 and t2 are lists of ints, and shift
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1052 is an int, chosen to minimize the combined size of t1 and t2 (in C
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1053 code), and where for each i in range(len(t)),
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1054 t[i] == t2[(t1[i >> shift] << shift) + (i & mask)]
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1055 where mask is a bitmask isolating the last "shift" bits.
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1056
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	1057 If optional arg trace is non-zero (default zero), progress info
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	1058 is printed to sys.stderr. The higher the value, the more info
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	1059 you'll get.
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1060 """
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1061
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1062 if trace:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1063 def dump(t1, t2, shift, bytes):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1064 print >>sys.stderr, "%d+%d bins at shift %d; %d bytes" % (
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1065 len(t1), len(t2), shift, bytes)
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1066 print >>sys.stderr, "Size of original table:", len(t)*getsize(t), \
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1067 "bytes"
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1068 n = len(t)-1 # last valid index
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1069 maxshift = 0 # the most we can shift n and still have something left
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1070 if n > 0:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1071 while n >> 1:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1072 n >>= 1
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1073 maxshift += 1
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1074 del n
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1075 bytes = sys.maxint # smallest total size so far
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1076 t = tuple(t) # so slices can be dict keys
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1077 for shift in range(maxshift + 1):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1078 t1 = []
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1079 t2 = []
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1080 size = 2**shift
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1081 bincache = {}
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1082 for i in range(0, len(t), size):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1083 bin = t[i:i+size]
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1084 index = bincache.get(bin)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1085 if index is None:
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1086 index = len(t2)
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1087 bincache[bin] = index
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1088 t2.extend(bin)
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1089 t1.append(index >> shift)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1090 # determine memory size
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1091 b = len(t1)getsize(t1) + len(t2)getsize(t2)
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	1092 if trace > 1:
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1093 dump(t1, t2, shift, b)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1094 if b < bytes:
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1095 best = t1, t2, shift
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1096 bytes = b
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1097 t1, t2, shift = best
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1098 if trace:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1099 print >>sys.stderr, "Best:",
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1100 dump(t1, t2, shift, bytes)
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1101 if __debug__:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1102 # exhaustively verify that the decomposition is correct
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1103 mask = ~((~0) << shift) # i.e., low-bit mask of shift bits
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1104 for i in xrange(len(t)):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1105 assert t[i] == t2[(t1[i >> shift] << shift) + (i & mask)]
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1106 return best
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1107
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1108 if __name__ == "__main__":
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	1109 maketables(1)

Mercurial > cpython-lru_cache-weakref

annotate Tools/unicode/makeunicodedata.py @ 58765:2ba7677bfa95 legacy-trunk