cpython-lru_cache-weakref: Tools/unicode/makeunicodedata.py annotate

annotate Tools/unicode/makeunicodedata.py @ 87690:66c1c9f32567 default tip

Add use_weakref option to functools.lru_cache The use_weakref option makes lru_cache create weak references to the function arguments when creating the cache key, which prevents them from staying in the memory forever even after all other references to them are gone. This is a proof of concept for http://bugs.python.org/issue19859

author	Radomir Dopieralski <devel@sheep.art.pl>
date	Tue, 03 Dec 2013 16:02:48 +0100
parents	97df7404f39c
children

rev	line source
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1 #
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	2 # (re)generate unicode property and type databases
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	3 #
26397 b27c1cad77ea Add unidata_version. Bump generator version number. Martin v. Löwis <martin@v.loewis.de> parents: 26394 diff changeset	4 # this script converts a unicode 3.2 database file to
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	5 # Modules/unicodedata_db.h, Modules/unicodename_db.h,
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	6 # and Objects/unicodetype_db.h
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	7 #
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	8 # history:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	9 # 2000-09-24 fl created (based on bits and pieces from unidb)
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	10 # 2000-09-25 fl merged tim's splitbin fixes, separate decomposition table
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	11 # 2000-09-25 fl added character type table
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	12 # 2000-09-26 fl added LINEBREAK, DECIMAL, and DIGIT flags/fields (2.0)
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	13 # 2000-11-03 fl expand first/last ranges
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	14 # 2001-01-19 fl added character name tables (2.1)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	15 # 2001-01-21 fl added decomp compression; dynamic phrasebook threshold
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	16 # 2002-09-11 wd use string methods
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	17 # 2002-10-18 mvl update to Unicode 3.2
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	18 # 2002-10-22 mvl generate NFC tables
26394 491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	19 # 2002-11-24 mvl expand all ranges, sort names version-independently
26397 b27c1cad77ea Add unidata_version. Bump generator version number. Martin v. Löwis <martin@v.loewis.de> parents: 26394 diff changeset	20 # 2002-11-25 mvl add UNIDATA_VERSION
32197 3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	21 # 2004-05-29 perky add east asian width information
36940 e4465d2737c4 Add changelog entry. Martin v. Löwis <martin@v.loewis.de> parents: 36937 diff changeset	22 # 2006-03-10 mvl update to Unicode 4.1; add UCD 3.2 delta
48427 4e6859c6d38c Issue #3282: str.isprintable() should return False for undefined Unicode characters. Georg Brandl <georg@python.org> parents: 47680 diff changeset	23 # 2008-06-11 gb add PRINTABLE_MASK for Atsuo Ishimoto's ascii() patch
73044 a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	24 # 2011-10-21 ezio add support for name aliases and named sequences
74338 f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	25 # 2012-01 benjamin add full case mappings
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	26 #
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	27 # written by Fredrik Lundh (fredrik@pythonware.com)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	28 #
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	29
73044 a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	30 import os
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	31 import sys
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	32 import zipfile
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	33
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	34 from textwrap import dedent
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	35
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	36 SCRIPT = sys.argv[0]
64146 8f97c64b2333 #5127: Even on narrow unicode builds, the C functions that access the Unicode Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 60086 diff changeset	37 VERSION = "3.2"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	38
26397 b27c1cad77ea Add unidata_version. Bump generator version number. Martin v. Löwis <martin@v.loewis.de> parents: 26394 diff changeset	39 # The Unicode Database
86200 97df7404f39c upgrade unicode db to 6.3.0 (closes #19221) Benjamin Peterson <benjamin@python.org> parents: 85407 diff changeset	40 UNIDATA_VERSION = "6.3.0"
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	41 UNICODE_DATA = "UnicodeData%s.txt"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	42 COMPOSITION_EXCLUSIONS = "CompositionExclusions%s.txt"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	43 EASTASIAN_WIDTH = "EastAsianWidth%s.txt"
65217 cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	44 UNIHAN = "Unihan%s.zip"
42595 f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	45 DERIVED_CORE_PROPERTIES = "DerivedCoreProperties%s.txt"
53554 423f8f9b07b1 Merged revisions 72054 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 53402 diff changeset	46 DERIVEDNORMALIZATION_PROPS = "DerivedNormalizationProps%s.txt"
60086 bf3c48f2417c Merged revisions 79494,79496 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 59720 diff changeset	47 LINE_BREAK = "LineBreak%s.txt"
73044 a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	48 NAME_ALIASES = "NameAliases%s.txt"
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	49 NAMED_SEQUENCES = "NamedSequences%s.txt"
74338 f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	50 SPECIAL_CASING = "SpecialCasing%s.txt"
74405 d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	51 CASE_FOLDING = "CaseFolding%s.txt"
73044 a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	52
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	53 # Private Use Areas -- in planes 1, 15, 16
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	54 PUA_1 = range(0xE000, 0xF900)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	55 PUA_15 = range(0xF0000, 0xFFFFE)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	56 PUA_16 = range(0x100000, 0x10FFFE)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	57
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	58 # we use this ranges of PUA_15 to store name aliases and named sequences
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	59 NAME_ALIASES_START = 0xF0000
75107 cd21e6d30a1a update to Unicode 6.1 Benjamin Peterson <benjamin@python.org> parents: 74423 diff changeset	60 NAMED_SEQUENCES_START = 0xF0200
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	61
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	62 old_versions = ["3.2.0"]
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	63
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	64 CATEGORY_NAMES = [ "Cn", "Lu", "Ll", "Lt", "Mn", "Mc", "Me", "Nd",
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	65 "Nl", "No", "Zs", "Zl", "Zp", "Cc", "Cf", "Cs", "Co", "Cn", "Lm",
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	66 "Lo", "Pc", "Pd", "Ps", "Pe", "Pi", "Pf", "Po", "Sm", "Sc", "Sk",
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	67 "So" ]
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	68
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	69 BIDIRECTIONAL_NAMES = [ "", "L", "LRE", "LRO", "R", "AL", "RLE", "RLO",
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	70 "PDF", "EN", "ES", "ET", "AN", "CS", "NSM", "BN", "B", "S", "WS",
86200 97df7404f39c upgrade unicode db to 6.3.0 (closes #19221) Benjamin Peterson <benjamin@python.org> parents: 85407 diff changeset	71 "ON", "LRI", "RLI", "FSI", "PDI" ]
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	72
32951 024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	73 EASTASIANWIDTH_NAMES = [ "F", "H", "W", "Na", "A", "N" ]
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	74
60086 bf3c48f2417c Merged revisions 79494,79496 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 59720 diff changeset	75 MANDATORY_LINE_BREAKS = [ "BK", "CR", "LF", "NL" ]
bf3c48f2417c Merged revisions 79494,79496 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 59720 diff changeset	76
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	77 # note: should match definitions in Objects/unicodectype.c
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	78 ALPHA_MASK = 0x01
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	79 DECIMAL_MASK = 0x02
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	80 DIGIT_MASK = 0x04
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	81 LOWER_MASK = 0x08
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	82 LINEBREAK_MASK = 0x10
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	83 SPACE_MASK = 0x20
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	84 TITLE_MASK = 0x40
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	85 UPPER_MASK = 0x80
42595 f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	86 XID_START_MASK = 0x100
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	87 XID_CONTINUE_MASK = 0x200
48427 4e6859c6d38c Issue #3282: str.isprintable() should return False for undefined Unicode characters. Georg Brandl <georg@python.org> parents: 47680 diff changeset	88 PRINTABLE_MASK = 0x400
74338 f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	89 NUMERIC_MASK = 0x800
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	90 CASE_IGNORABLE_MASK = 0x1000
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	91 CASED_MASK = 0x2000
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	92 EXTENDED_CASE_MASK = 0x4000
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	93
66392 d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	94 # these ranges need to match unicodedata.c:is_unified_ideograph
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	95 cjk_ranges = [
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	96 ('3400', '4DB5'),
75107 cd21e6d30a1a update to Unicode 6.1 Benjamin Peterson <benjamin@python.org> parents: 74423 diff changeset	97 ('4E00', '9FCC'),
66392 d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	98 ('20000', '2A6D6'),
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	99 ('2A700', '2B734'),
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	100 ('2B740', '2B81D')
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	101 ]
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	102
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	103 def maketables(trace=0):
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	104
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	105 print("--- Reading", UNICODE_DATA % "", "...")
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	106
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	107 version = ""
65217 cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	108 unicode = UnicodeData(UNIDATA_VERSION)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	109
47611 da47e7e135ae #2630: Implement PEP 3138. Georg Brandl <georg@python.org> parents: 47463 diff changeset	110 print(len(list(filter(None, unicode.table))), "characters")
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	111
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	112 for version in old_versions:
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	113 print("--- Reading", UNICODE_DATA % ("-"+version), "...")
66392 d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	114 old_unicode = UnicodeData(version, cjk_check=False)
47611 da47e7e135ae #2630: Implement PEP 3138. Georg Brandl <georg@python.org> parents: 47463 diff changeset	115 print(len(list(filter(None, old_unicode.table))), "characters")
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	116 merge_old_version(version, unicode, old_unicode)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	117
16091 45c065668ea5 Unicode nits: Don't include unicodedatabase.h no more. And make sure Fredrik Lundh <fredrik@pythonware.com> parents: 16090 diff changeset	118 makeunicodename(unicode, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	119 makeunicodedata(unicode, trace)
16091 45c065668ea5 Unicode nits: Don't include unicodedatabase.h no more. And make sure Fredrik Lundh <fredrik@pythonware.com> parents: 16090 diff changeset	120 makeunicodetype(unicode, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	121
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	122 # --------------------------------------------------------------------
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	123 # unicode character properties
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	124
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	125 def makeunicodedata(unicode, trace):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	126
53554 423f8f9b07b1 Merged revisions 72054 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 53402 diff changeset	127 dummy = (0, 0, 0, 0, 0, 0)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	128 table = [dummy]
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	129 cache = {0: dummy}
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	130 index = [0] * len(unicode.chars)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	131
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	132 FILE = "Modules/unicodedata_db.h"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	133
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	134 print("--- Preparing", FILE, "...")
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	135
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	136 # 1) database properties
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	137
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	138 for char in unicode.chars:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	139 record = unicode.table[char]
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	140 if record:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	141 # extract database properties
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	142 category = CATEGORY_NAMES.index(record[2])
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	143 combining = int(record[3])
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	144 bidirectional = BIDIRECTIONAL_NAMES.index(record[4])
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	145 mirrored = record[9] == "Y"
32951 024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	146 eastasianwidth = EASTASIANWIDTH_NAMES.index(record[15])
53554 423f8f9b07b1 Merged revisions 72054 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 53402 diff changeset	147 normalizationquickcheck = record[17]
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	148 item = (
53554 423f8f9b07b1 Merged revisions 72054 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 53402 diff changeset	149 category, combining, bidirectional, mirrored, eastasianwidth,
423f8f9b07b1 Merged revisions 72054 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 53402 diff changeset	150 normalizationquickcheck
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	151 )
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	152 # add entry to index and item tables
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	153 i = cache.get(item)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	154 if i is None:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	155 cache[item] = i = len(table)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	156 table.append(item)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	157 index[char] = i
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	158
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	159 # 2) decomposition data
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	160
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	161 decomp_data = [0]
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	162 decomp_prefix = [""]
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	163 decomp_index = [0] * len(unicode.chars)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	164 decomp_size = 0
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	165
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	166 comp_pairs = []
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	167 comp_first = [None] * len(unicode.chars)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	168 comp_last = [None] * len(unicode.chars)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	169
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	170 for char in unicode.chars:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	171 record = unicode.table[char]
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	172 if record:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	173 if record[5]:
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	174 decomp = record[5].split()
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	175 if len(decomp) > 19:
42715 e4ade16a694f Convert raise statements in Tools/. Collin Winter <collinw@gmail.com> parents: 42595 diff changeset	176 raise Exception("character %x has a decomposition too large for nfd_nfkd" % char)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	177 # prefix
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	178 if decomp[0][0] == "<":
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	179 prefix = decomp.pop(0)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	180 else:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	181 prefix = ""
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	182 try:
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	183 i = decomp_prefix.index(prefix)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	184 except ValueError:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	185 i = len(decomp_prefix)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	186 decomp_prefix.append(prefix)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	187 prefix = i
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	188 assert prefix < 256
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	189 # content
47103 63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893. Georg Brandl <georg@python.org> parents: 45049 diff changeset	190 decomp = [prefix + (len(decomp)<<8)] + [int(s, 16) for s in decomp]
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	191 # Collect NFC pairs
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	192 if not prefix and len(decomp) == 3 and \
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	193 char not in unicode.exclusions and \
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	194 unicode.table[decomp[1]][3] == "0":
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	195 p, l, r = decomp
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	196 comp_first[l] = 1
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	197 comp_last[r] = 1
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	198 comp_pairs.append((l,r,char))
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	199 try:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	200 i = decomp_data.index(decomp)
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	201 except ValueError:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	202 i = len(decomp_data)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	203 decomp_data.extend(decomp)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	204 decomp_size = decomp_size + len(decomp) * 2
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	205 else:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	206 i = 0
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	207 decomp_index[char] = i
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	208
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	209 f = l = 0
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	210 comp_first_ranges = []
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	211 comp_last_ranges = []
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	212 prev_f = prev_l = None
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	213 for i in unicode.chars:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	214 if comp_first[i] is not None:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	215 comp_first[i] = f
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	216 f += 1
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	217 if prev_f is None:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	218 prev_f = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	219 elif prev_f[1]+1 == i:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	220 prev_f = prev_f[0],i
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	221 else:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	222 comp_first_ranges.append(prev_f)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	223 prev_f = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	224 if comp_last[i] is not None:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	225 comp_last[i] = l
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	226 l += 1
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	227 if prev_l is None:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	228 prev_l = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	229 elif prev_l[1]+1 == i:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	230 prev_l = prev_l[0],i
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	231 else:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	232 comp_last_ranges.append(prev_l)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	233 prev_l = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	234 comp_first_ranges.append(prev_f)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	235 comp_last_ranges.append(prev_l)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	236 total_first = f
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	237 total_last = l
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	238
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	239 comp_data = [0](total_firsttotal_last)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	240 for f,l,char in comp_pairs:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	241 f = comp_first[f]
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	242 l = comp_last[l]
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	243 comp_data[f*total_last+l] = char
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	244
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	245 print(len(table), "unique properties")
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	246 print(len(decomp_prefix), "unique decomposition prefixes")
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	247 print(len(decomp_data), "unique decomposition entries:", end=' ')
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	248 print(decomp_size, "bytes")
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	249 print(total_first, "first characters in NFC")
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	250 print(total_last, "last characters in NFC")
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	251 print(len(comp_pairs), "NFC pairs")
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	252
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	253 print("--- Writing", FILE, "...")
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	254
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	255 fp = open(FILE, "w")
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	256 print("/* this file was generated by %s %s */" % (SCRIPT, VERSION), file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	257 print(file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	258 print('#define UNIDATA_VERSION "%s"' % UNIDATA_VERSION, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	259 print("/* a list of unique database records */", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	260 print("const _PyUnicode_DatabaseRecord _PyUnicode_Database_Records[] = {", file=fp)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	261 for item in table:
53554 423f8f9b07b1 Merged revisions 72054 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 53402 diff changeset	262 print(" {%d, %d, %d, %d, %d, %d}," % item, file=fp)
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	263 print("};", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	264 print(file=fp)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	265
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	266 print("/* Reindexing of NFC first characters. */", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	267 print("#define TOTAL_FIRST",total_first, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	268 print("#define TOTAL_LAST",total_last, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	269 print("struct reindex{int start;short count,index;};", file=fp)
47680 b5e02775fffc Merged revisions 64226 via svnmerge from Martin v. Löwis <martin@v.loewis.de> parents: 47611 diff changeset	270 print("static struct reindex nfc_first[] = {", file=fp)
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	271 for start,end in comp_first_ranges:
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	272 print(" { %d, %d, %d}," % (start,end-start,comp_first[start]), file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	273 print(" {0,0,0}", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	274 print("};\n", file=fp)
47680 b5e02775fffc Merged revisions 64226 via svnmerge from Martin v. Löwis <martin@v.loewis.de> parents: 47611 diff changeset	275 print("static struct reindex nfc_last[] = {", file=fp)
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	276 for start,end in comp_last_ranges:
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	277 print(" { %d, %d, %d}," % (start,end-start,comp_last[start]), file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	278 print(" {0,0,0}", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	279 print("};\n", file=fp)
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	280
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	281 # FIXME: <fl> the following tables could be made static, and
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	282 # the support code moved into unicodedatabase.c
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	283
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	284 print("/* string literals */", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	285 print("const char *_PyUnicode_CategoryNames[] = {", file=fp)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	286 for name in CATEGORY_NAMES:
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	287 print(" \"%s\"," % name, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	288 print(" NULL", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	289 print("};", file=fp)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	290
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	291 print("const char *_PyUnicode_BidirectionalNames[] = {", file=fp)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	292 for name in BIDIRECTIONAL_NAMES:
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	293 print(" \"%s\"," % name, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	294 print(" NULL", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	295 print("};", file=fp)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	296
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	297 print("const char *_PyUnicode_EastAsianWidthNames[] = {", file=fp)
32951 024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	298 for name in EASTASIANWIDTH_NAMES:
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	299 print(" \"%s\"," % name, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	300 print(" NULL", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	301 print("};", file=fp)
32951 024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	302
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	303 print("static const char *decomp_prefix[] = {", file=fp)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	304 for name in decomp_prefix:
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	305 print(" \"%s\"," % name, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	306 print(" NULL", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	307 print("};", file=fp)
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	308
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	309 # split record index table
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	310 index1, index2, shift = splitbins(index, trace)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	311
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	312 print("/* index tables for the database records */", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	313 print("#define SHIFT", shift, file=fp)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	314 Array("index1", index1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	315 Array("index2", index2).dump(fp, trace)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	316
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	317 # split decomposition index table
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	318 index1, index2, shift = splitbins(decomp_index, trace)
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	319
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	320 print("/* decomposition data */", file=fp)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	321 Array("decomp_data", decomp_data).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	322
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	323 print("/* index tables for the decomposition data */", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	324 print("#define DECOMP_SHIFT", shift, file=fp)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	325 Array("decomp_index1", index1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	326 Array("decomp_index2", index2).dump(fp, trace)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	327
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	328 index, index2, shift = splitbins(comp_data, trace)
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	329 print("/* NFC pairs */", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	330 print("#define COMP_SHIFT", shift, file=fp)
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	331 Array("comp_index", index).dump(fp, trace)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	332 Array("comp_data", index2).dump(fp, trace)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	333
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	334 # Generate delta tables for old versions
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	335 for version, table, normalization in unicode.changed:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	336 cversion = version.replace(".","_")
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	337 records = [table[0]]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	338 cache = {table[0]:0}
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	339 index = [0] * len(table)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	340 for i, record in enumerate(table):
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	341 try:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	342 index[i] = cache[record]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	343 except KeyError:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	344 index[i] = cache[record] = len(records)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	345 records.append(record)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	346 index1, index2, shift = splitbins(index, trace)
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	347 print("static const change_record change_records_%s[] = {" % cversion, file=fp)
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	348 for record in records:
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	349 print("\t{ %s }," % ", ".join(map(str,record)), file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	350 print("};", file=fp)
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	351 Array("changes_%s_index" % cversion, index1).dump(fp, trace)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	352 Array("changes_%s_data" % cversion, index2).dump(fp, trace)
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	353 print("static const change_record* get_change_%s(Py_UCS4 n)" % cversion, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	354 print("{", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	355 print("\tint index;", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	356 print("\tif (n >= 0x110000) index = 0;", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	357 print("\telse {", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	358 print("\t\tindex = changes_%s_index[n>>%d];" % (cversion, shift), file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	359 print("\t\tindex = changes_%s_data[(index<<%d)+(n & %d)];" % \
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	360 (cversion, shift, ((1<<shift)-1)), file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	361 print("\t}", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	362 print("\treturn change_records_%s+index;" % cversion, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	363 print("}\n", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	364 print("static Py_UCS4 normalization_%s(Py_UCS4 n)" % cversion, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	365 print("{", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	366 print("\tswitch(n) {", file=fp)
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	367 for k, v in normalization:
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	368 print("\tcase %s: return 0x%s;" % (hex(k), v), file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	369 print("\tdefault: return 0;", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	370 print("\t}\n}\n", file=fp)
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	371
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	372 fp.close()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	373
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	374 # --------------------------------------------------------------------
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	375 # unicode character type tables
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	376
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	377 def makeunicodetype(unicode, trace):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	378
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	379 FILE = "Objects/unicodetype_db.h"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	380
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	381 print("--- Preparing", FILE, "...")
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	382
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	383 # extract unicode types
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	384 dummy = (0, 0, 0, 0, 0, 0)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	385 table = [dummy]
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	386 cache = {0: dummy}
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	387 index = [0] * len(unicode.chars)
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	388 numeric = {}
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	389 spaces = []
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	390 linebreaks = []
74338 f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	391 extra_casing = []
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	392
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	393 for char in unicode.chars:
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	394 record = unicode.table[char]
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	395 if record:
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	396 # extract database properties
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	397 category = record[2]
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	398 bidirectional = record[4]
42595 f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	399 properties = record[16]
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	400 flags = 0
49125 dc60293b29d5 Merged revisions 66362 via svnmerge from Martin v. Löwis <martin@v.loewis.de> parents: 48427 diff changeset	401 delta = True
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	402 if category in ["Lm", "Lt", "Lu", "Ll", "Lo"]:
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	403 flags \|= ALPHA_MASK
74338 f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	404 if "Lowercase" in properties:
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	405 flags \|= LOWER_MASK
60086 bf3c48f2417c Merged revisions 79494,79496 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 59720 diff changeset	406 if 'Line_Break' in properties or bidirectional == "B":
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	407 flags \|= LINEBREAK_MASK
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	408 linebreaks.append(char)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	409 if category == "Zs" or bidirectional in ("WS", "B", "S"):
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	410 flags \|= SPACE_MASK
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	411 spaces.append(char)
14856 83aff1ea5b70 - don't set the titlecase flag for uppercase letters (sorry, tim) Fredrik Lundh <fredrik@pythonware.com> parents: 14852 diff changeset	412 if category == "Lt":
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	413 flags \|= TITLE_MASK
74338 f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	414 if "Uppercase" in properties:
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	415 flags \|= UPPER_MASK
52364 ca02bd8c96f2 Merged revisions 70610 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 51122 diff changeset	416 if char == ord(" ") or category[0] not in ("C", "Z"):
48427 4e6859c6d38c Issue #3282: str.isprintable() should return False for undefined Unicode characters. Georg Brandl <georg@python.org> parents: 47680 diff changeset	417 flags \|= PRINTABLE_MASK
42595 f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	418 if "XID_Start" in properties:
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	419 flags \|= XID_START_MASK
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	420 if "XID_Continue" in properties:
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	421 flags \|= XID_CONTINUE_MASK
74338 f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	422 if "Cased" in properties:
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	423 flags \|= CASED_MASK
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	424 if "Case_Ignorable" in properties:
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	425 flags \|= CASE_IGNORABLE_MASK
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	426 sc = unicode.special_casing.get(char)
74405 d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	427 cf = unicode.case_folding.get(char, [char])
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	428 if record[12]:
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	429 upper = int(record[12], 16)
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	430 else:
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	431 upper = char
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	432 if record[13]:
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	433 lower = int(record[13], 16)
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	434 else:
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	435 lower = char
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	436 if record[14]:
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	437 title = int(record[14], 16)
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	438 else:
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	439 title = upper
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	440 if sc is None and cf != [lower]:
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	441 sc = ([lower], [title], [upper])
74338 f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	442 if sc is None:
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	443 if upper == lower == title:
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	444 upper = lower = title = 0
74423 03ea95e3b497 delta encoding of upper/lower/title makes a glorious return (#12736) Benjamin Peterson <benjamin@python.org> parents: 74405 diff changeset	445 else:
03ea95e3b497 delta encoding of upper/lower/title makes a glorious return (#12736) Benjamin Peterson <benjamin@python.org> parents: 74405 diff changeset	446 upper = upper - char
03ea95e3b497 delta encoding of upper/lower/title makes a glorious return (#12736) Benjamin Peterson <benjamin@python.org> parents: 74405 diff changeset	447 lower = lower - char
03ea95e3b497 delta encoding of upper/lower/title makes a glorious return (#12736) Benjamin Peterson <benjamin@python.org> parents: 74405 diff changeset	448 title = title - char
03ea95e3b497 delta encoding of upper/lower/title makes a glorious return (#12736) Benjamin Peterson <benjamin@python.org> parents: 74405 diff changeset	449 assert (abs(upper) <= 2147483647 and
03ea95e3b497 delta encoding of upper/lower/title makes a glorious return (#12736) Benjamin Peterson <benjamin@python.org> parents: 74405 diff changeset	450 abs(lower) <= 2147483647 and
03ea95e3b497 delta encoding of upper/lower/title makes a glorious return (#12736) Benjamin Peterson <benjamin@python.org> parents: 74405 diff changeset	451 abs(title) <= 2147483647)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	452 else:
74405 d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	453 # This happens either when some character maps to more than one
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	454 # character in uppercase, lowercase, or titlecase or the
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	455 # casefolded version of the character is different from the
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	456 # lowercase. The extra characters are stored in a different
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	457 # array.
74338 f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	458 flags \|= EXTENDED_CASE_MASK
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	459 lower = len(extra_casing) \| (len(sc[0]) << 24)
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	460 extra_casing.extend(sc[0])
74405 d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	461 if cf != sc[0]:
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	462 lower \|= len(cf) << 20
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	463 extra_casing.extend(cf)
74338 f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	464 upper = len(extra_casing) \| (len(sc[2]) << 24)
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	465 extra_casing.extend(sc[2])
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	466 # Title is probably equal to upper.
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	467 if sc[1] == sc[2]:
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	468 title = upper
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	469 else:
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	470 title = len(extra_casing) \| (len(sc[1]) << 24)
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	471 extra_casing.extend(sc[1])
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	472 # decimal digit, integer digit
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	473 decimal = 0
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	474 if record[6]:
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	475 flags \|= DECIMAL_MASK
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	476 decimal = int(record[6])
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	477 digit = 0
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	478 if record[7]:
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	479 flags \|= DIGIT_MASK
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	480 digit = int(record[7])
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	481 if record[8]:
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	482 flags \|= NUMERIC_MASK
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	483 numeric.setdefault(record[8], []).append(char)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	484 item = (
32197 3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	485 upper, lower, title, decimal, digit, flags
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	486 )
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	487 # add entry to index and item tables
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	488 i = cache.get(item)
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	489 if i is None:
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	490 cache[item] = i = len(table)
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	491 table.append(item)
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	492 index[char] = i
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	493
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	494 print(len(table), "unique character type entries")
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	495 print(sum(map(len, numeric.values())), "numeric code points")
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	496 print(len(spaces), "whitespace code points")
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	497 print(len(linebreaks), "linebreak code points")
74338 f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	498 print(len(extra_casing), "extended case array")
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	499
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	500 print("--- Writing", FILE, "...")
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	501
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	502 fp = open(FILE, "w")
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	503 print("/* this file was generated by %s %s */" % (SCRIPT, VERSION), file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	504 print(file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	505 print("/* a list of unique character type descriptors */", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	506 print("const _PyUnicode_TypeRecord _PyUnicode_TypeRecords[] = {", file=fp)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	507 for item in table:
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	508 print(" {%d, %d, %d, %d, %d, %d}," % item, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	509 print("};", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	510 print(file=fp)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	511
74338 f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	512 print("/* extended case mappings */", file=fp)
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	513 print(file=fp)
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	514 print("const Py_UCS4 _PyUnicode_ExtendedCase[] = {", file=fp)
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	515 for c in extra_casing:
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	516 print(" %d," % c, file=fp)
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	517 print("};", file=fp)
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	518 print(file=fp)
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	519
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	520 # split decomposition index table
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	521 index1, index2, shift = splitbins(index, trace)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	522
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	523 print("/* type indexes */", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	524 print("#define SHIFT", shift, file=fp)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	525 Array("index1", index1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	526 Array("index2", index2).dump(fp, trace)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	527
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	528 # Generate code for _PyUnicode_ToNumeric()
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	529 numeric_items = sorted(numeric.items())
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	530 print('/* Returns the numeric value as double for Unicode characters', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	531 print(' * having this property, -1.0 otherwise.', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	532 print(' */', file=fp)
64146 8f97c64b2333 #5127: Even on narrow unicode builds, the C functions that access the Unicode Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 60086 diff changeset	533 print('double _PyUnicode_ToNumeric(Py_UCS4 ch)', file=fp)
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	534 print('{', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	535 print(' switch (ch) {', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	536 for value, codepoints in numeric_items:
56508 e8e103488fb3 Merged revisions 75396 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 56408 diff changeset	537 # Turn text into float literals
e8e103488fb3 Merged revisions 75396 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 56408 diff changeset	538 parts = value.split('/')
e8e103488fb3 Merged revisions 75396 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 56408 diff changeset	539 parts = [repr(float(part)) for part in parts]
e8e103488fb3 Merged revisions 75396 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 56408 diff changeset	540 value = '/'.join(parts)
e8e103488fb3 Merged revisions 75396 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 56408 diff changeset	541
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	542 codepoints.sort()
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	543 for codepoint in codepoints:
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	544 print(' case 0x%04X:' % (codepoint,), file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	545 print(' return (double) %s;' % (value,), file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	546 print(' }', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	547 print(' return -1.0;', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	548 print('}', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	549 print(file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	550
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	551 # Generate code for _PyUnicode_IsWhitespace()
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	552 print("/* Returns 1 for Unicode characters having the bidirectional", file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	553 print(" * type 'WS', 'B' or 'S' or the category 'Zs', 0 otherwise.", file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	554 print(" */", file=fp)
85157 e7f6cef7a4cc Issue #18722: Remove uses of the "register" keyword in C code. Antoine Pitrou <solipsis@pitrou.net> parents: 79255 diff changeset	555 print('int _PyUnicode_IsWhitespace(const Py_UCS4 ch)', file=fp)
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	556 print('{', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	557 print(' switch (ch) {', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	558
59720 8a28be45c242 Merged revisions 78982,78986 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 59719 diff changeset	559 for codepoint in sorted(spaces):
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	560 print(' case 0x%04X:' % (codepoint,), file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	561 print(' return 1;', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	562
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	563 print(' }', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	564 print(' return 0;', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	565 print('}', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	566 print(file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	567
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	568 # Generate code for _PyUnicode_IsLinebreak()
60086 bf3c48f2417c Merged revisions 79494,79496 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 59720 diff changeset	569 print("/* Returns 1 for Unicode characters having the line break", file=fp)
bf3c48f2417c Merged revisions 79494,79496 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 59720 diff changeset	570 print(" * property 'BK', 'CR', 'LF' or 'NL' or having bidirectional", file=fp)
bf3c48f2417c Merged revisions 79494,79496 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 59720 diff changeset	571 print(" * type 'B', 0 otherwise.", file=fp)
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	572 print(" */", file=fp)
85157 e7f6cef7a4cc Issue #18722: Remove uses of the "register" keyword in C code. Antoine Pitrou <solipsis@pitrou.net> parents: 79255 diff changeset	573 print('int _PyUnicode_IsLinebreak(const Py_UCS4 ch)', file=fp)
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	574 print('{', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	575 print(' switch (ch) {', file=fp)
59720 8a28be45c242 Merged revisions 78982,78986 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 59719 diff changeset	576 for codepoint in sorted(linebreaks):
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	577 print(' case 0x%04X:' % (codepoint,), file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	578 print(' return 1;', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	579
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	580 print(' }', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	581 print(' return 0;', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	582 print('}', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	583 print(file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	584
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	585 fp.close()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	586
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	587 # --------------------------------------------------------------------
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	588 # unicode name database
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	589
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	590 def makeunicodename(unicode, trace):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	591
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	592 FILE = "Modules/unicodename_db.h"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	593
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	594 print("--- Preparing", FILE, "...")
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	595
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	596 # collect names
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	597 names = [None] * len(unicode.chars)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	598
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	599 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	600 record = unicode.table[char]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	601 if record:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	602 name = record[1].strip()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	603 if name and name[0] != "<":
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	604 names[char] = name + chr(0)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	605
47611 da47e7e135ae #2630: Implement PEP 3138. Georg Brandl <georg@python.org> parents: 47463 diff changeset	606 print(len(list(n for n in names if n is not None)), "distinct names")
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	607
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	608 # collect unique words from names (note that we differ between
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	609 # words inside a sentence, and words ending a sentence. the
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	610 # latter includes the trailing null byte.
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	611
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	612 words = {}
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	613 n = b = 0
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	614 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	615 name = names[char]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	616 if name:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	617 w = name.split()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	618 b = b + len(name)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	619 n = n + len(w)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	620 for w in w:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	621 l = words.get(w)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	622 if l:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	623 l.append(None)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	624 else:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	625 words[w] = [len(words)]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	626
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	627 print(n, "words in text;", b, "bytes")
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	628
42595 f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	629 wordlist = list(words.items())
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	630
26394 491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	631 # sort on falling frequency, then by name
51122 adbd77a90bde Merged revisions 69025 via svnmerge from Mark Dickinson <dickinsm@gmail.com> parents: 49125 diff changeset	632 def word_key(a):
42595 f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	633 aword, alist = a
51122 adbd77a90bde Merged revisions 69025 via svnmerge from Mark Dickinson <dickinsm@gmail.com> parents: 49125 diff changeset	634 return -len(alist), aword
adbd77a90bde Merged revisions 69025 via svnmerge from Mark Dickinson <dickinsm@gmail.com> parents: 49125 diff changeset	635 wordlist.sort(key=word_key)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	636
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	637 # figure out how many phrasebook escapes we need
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	638 escapes = 0
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	639 while escapes * 256 < len(wordlist):
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	640 escapes = escapes + 1
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	641 print(escapes, "escapes")
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	642
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	643 short = 256 - escapes
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	644
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	645 assert short > 0
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	646
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	647 print(short, "short indexes in lexicon")
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	648
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	649 # statistics
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	650 n = 0
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	651 for i in range(short):
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	652 n = n + len(wordlist[i][1])
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	653 print(n, "short indexes in phrasebook")
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	654
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	655 # pick the most commonly used words, and sort the rest on falling
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	656 # length (to maximize overlap)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	657
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	658 wordlist, wordtail = wordlist[:short], wordlist[short:]
45049 90e86ee3684b Convert some custom sort comparison functions to equivalent key functions. Raymond Hettinger <python@rcn.com> parents: 44160 diff changeset	659 wordtail.sort(key=lambda a: a[0], reverse=True)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	660 wordlist.extend(wordtail)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	661
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	662 # generate lexicon from words
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	663
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	664 lexicon_offset = [0]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	665 lexicon = ""
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	666 words = {}
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	667
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	668 # build a lexicon string
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	669 offset = 0
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	670 for w, x in wordlist:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	671 # encoding: bit 7 indicates last character in word (chr(128)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	672 # indicates the last character in an entire string)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	673 ww = w[:-1] + chr(ord(w[-1])+128)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	674 # reuse string tails, when possible
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	675 o = lexicon.find(ww)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	676 if o < 0:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	677 o = offset
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	678 lexicon = lexicon + ww
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	679 offset = offset + len(w)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	680 words[w] = len(lexicon_offset)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	681 lexicon_offset.append(o)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	682
42595 f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	683 lexicon = list(map(ord, lexicon))
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	684
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	685 # generate phrasebook from names and lexicon
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	686 phrasebook = [0]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	687 phrasebook_offset = [0] * len(unicode.chars)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	688 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	689 name = names[char]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	690 if name:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	691 w = name.split()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	692 phrasebook_offset[char] = len(phrasebook)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	693 for w in w:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	694 i = words[w]
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	695 if i < short:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	696 phrasebook.append(i)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	697 else:
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	698 # store as two bytes
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	699 phrasebook.append((i>>8) + short)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	700 phrasebook.append(i&255)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	701
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	702 assert getsize(phrasebook) == 1
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	703
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	704 #
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	705 # unicode name hash table
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	706
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	707 # extract names
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	708 data = []
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	709 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	710 record = unicode.table[char]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	711 if record:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	712 name = record[1].strip()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	713 if name and name[0] != "<":
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	714 data.append((name, char))
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	715
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	716 # the magic number 47 was chosen to minimize the number of
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	717 # collisions on the current data set. if you like, change it
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	718 # and see what happens...
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	719
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	720 codehash = Hash("code", data, 47)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	721
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	722 print("--- Writing", FILE, "...")
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	723
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	724 fp = open(FILE, "w")
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	725 print("/* this file was generated by %s %s */" % (SCRIPT, VERSION), file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	726 print(file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	727 print("#define NAME_MAXLEN", 256, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	728 print(file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	729 print("/* lexicon */", file=fp)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	730 Array("lexicon", lexicon).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	731 Array("lexicon_offset", lexicon_offset).dump(fp, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	732
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	733 # split decomposition index table
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	734 offset1, offset2, shift = splitbins(phrasebook_offset, trace)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	735
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	736 print("/* code->name phrasebook */", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	737 print("#define phrasebook_shift", shift, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	738 print("#define phrasebook_short", short, file=fp)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	739
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	740 Array("phrasebook", phrasebook).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	741 Array("phrasebook_offset1", offset1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	742 Array("phrasebook_offset2", offset2).dump(fp, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	743
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	744 print("/* name->code dictionary */", file=fp)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	745 codehash.dump(fp, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	746
73044 a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	747 print(file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	748 print('static const unsigned int aliases_start = %#x;' %
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	749 NAME_ALIASES_START, file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	750 print('static const unsigned int aliases_end = %#x;' %
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	751 (NAME_ALIASES_START + len(unicode.aliases)), file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	752
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	753 print('static const unsigned int name_aliases[] = {', file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	754 for name, codepoint in unicode.aliases:
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	755 print(' 0x%04X,' % codepoint, file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	756 print('};', file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	757
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	758 # In Unicode 6.0.0, the sequences contain at most 4 BMP chars,
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	759 # so we are using Py_UCS2 seq[4]. This needs to be updated if longer
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	760 # sequences or sequences with non-BMP chars are added.
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	761 # unicodedata_lookup should be adapted too.
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	762 print(dedent("""
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	763 typedef struct NamedSequence {
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	764 int seqlen;
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	765 Py_UCS2 seq[4];
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	766 } named_sequence;
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	767 """), file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	768
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	769 print('static const unsigned int named_sequences_start = %#x;' %
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	770 NAMED_SEQUENCES_START, file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	771 print('static const unsigned int named_sequences_end = %#x;' %
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	772 (NAMED_SEQUENCES_START + len(unicode.named_sequences)), file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	773
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	774 print('static const named_sequence named_sequences[] = {', file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	775 for name, sequence in unicode.named_sequences:
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	776 seq_str = ', '.join('0x%04X' % cp for cp in sequence)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	777 print(' {%d, {%s}},' % (len(sequence), seq_str), file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	778 print('};', file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	779
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	780 fp.close()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	781
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	782
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	783 def merge_old_version(version, new, old):
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	784 # Changes to exclusion file not implemented yet
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	785 if old.exclusions != new.exclusions:
42715 e4ade16a694f Convert raise statements in Tools/. Collin Winter <collinw@gmail.com> parents: 42595 diff changeset	786 raise NotImplementedError("exclusions differ")
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	787
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	788 # In these change records, 0xFF means "no change"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	789 bidir_changes = [0xFF]*0x110000
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	790 category_changes = [0xFF]*0x110000
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	791 decimal_changes = [0xFF]*0x110000
49125 dc60293b29d5 Merged revisions 66362 via svnmerge from Martin v. Löwis <martin@v.loewis.de> parents: 48427 diff changeset	792 mirrored_changes = [0xFF]*0x110000
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	793 # In numeric data, 0 means "no change",
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	794 # -1 means "did not have a numeric value
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	795 numeric_changes = [0] * 0x110000
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	796 # normalization_changes is a list of key-value pairs
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	797 normalization_changes = []
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	798 for i in range(0x110000):
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	799 if new.table[i] is None:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	800 # Characters unassigned in the new version ought to
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	801 # be unassigned in the old one
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	802 assert old.table[i] is None
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	803 continue
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	804 # check characters unassigned in the old version
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	805 if old.table[i] is None:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	806 # category 0 is "unassigned"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	807 category_changes[i] = 0
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	808 continue
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	809 # check characters that differ
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	810 if old.table[i] != new.table[i]:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	811 for k in range(len(old.table[i])):
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	812 if old.table[i][k] != new.table[i][k]:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	813 value = old.table[i][k]
73044 a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	814 if k == 1 and i in PUA_15:
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	815 # the name is not set in the old.table, but in the
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	816 # new.table we are using it for aliases and named seq
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	817 assert value == ''
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	818 elif k == 2:
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	819 #print "CATEGORY",hex(i), old.table[i][k], new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	820 category_changes[i] = CATEGORY_NAMES.index(value)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	821 elif k == 4:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	822 #print "BIDIR",hex(i), old.table[i][k], new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	823 bidir_changes[i] = BIDIRECTIONAL_NAMES.index(value)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	824 elif k == 5:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	825 #print "DECOMP",hex(i), old.table[i][k], new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	826 # We assume that all normalization changes are in 1:1 mappings
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	827 assert " " not in value
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	828 normalization_changes.append((i, value))
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	829 elif k == 6:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	830 #print "DECIMAL",hex(i), old.table[i][k], new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	831 # we only support changes where the old value is a single digit
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	832 assert value in "0123456789"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	833 decimal_changes[i] = int(value)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	834 elif k == 8:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	835 # print "NUMERIC",hex(i), `old.table[i][k]`, new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	836 # Since 0 encodes "no change", the old value is better not 0
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	837 if not value:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	838 numeric_changes[i] = -1
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	839 else:
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	840 numeric_changes[i] = float(value)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	841 assert numeric_changes[i] not in (0, -1)
49125 dc60293b29d5 Merged revisions 66362 via svnmerge from Martin v. Löwis <martin@v.loewis.de> parents: 48427 diff changeset	842 elif k == 9:
dc60293b29d5 Merged revisions 66362 via svnmerge from Martin v. Löwis <martin@v.loewis.de> parents: 48427 diff changeset	843 if value == 'Y':
dc60293b29d5 Merged revisions 66362 via svnmerge from Martin v. Löwis <martin@v.loewis.de> parents: 48427 diff changeset	844 mirrored_changes[i] = '1'
dc60293b29d5 Merged revisions 66362 via svnmerge from Martin v. Löwis <martin@v.loewis.de> parents: 48427 diff changeset	845 else:
dc60293b29d5 Merged revisions 66362 via svnmerge from Martin v. Löwis <martin@v.loewis.de> parents: 48427 diff changeset	846 mirrored_changes[i] = '0'
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	847 elif k == 11:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	848 # change to ISO comment, ignore
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	849 pass
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	850 elif k == 12:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	851 # change to simple uppercase mapping; ignore
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	852 pass
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	853 elif k == 13:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	854 # change to simple lowercase mapping; ignore
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	855 pass
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	856 elif k == 14:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	857 # change to simple titlecase mapping; ignore
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	858 pass
42595 f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	859 elif k == 16:
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	860 # derived property changes; not yet
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	861 pass
65217 cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	862 elif k == 17:
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	863 # normalization quickchecks are not performed
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	864 # for older versions
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	865 pass
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	866 else:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	867 class Difference(Exception):pass
42715 e4ade16a694f Convert raise statements in Tools/. Collin Winter <collinw@gmail.com> parents: 42595 diff changeset	868 raise Difference(hex(i), k, old.table[i], new.table[i])
42595 f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	869 new.changed.append((version, list(zip(bidir_changes, category_changes,
49125 dc60293b29d5 Merged revisions 66362 via svnmerge from Martin v. Löwis <martin@v.loewis.de> parents: 48427 diff changeset	870 decimal_changes, mirrored_changes,
dc60293b29d5 Merged revisions 66362 via svnmerge from Martin v. Löwis <martin@v.loewis.de> parents: 48427 diff changeset	871 numeric_changes)),
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	872 normalization_changes))
36937 df43c04d2554 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 36924 diff changeset	873
65217 cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	874 def open_data(template, version):
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	875 local = template % ('-'+version,)
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	876 if not os.path.exists(local):
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	877 import urllib.request
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	878 if version == '3.2.0':
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	879 # irregular url structure
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	880 url = 'http://www.unicode.org/Public/3.2-Update/' + local
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	881 else:
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	882 url = ('http://www.unicode.org/Public/%s/ucd/'+template) % (version, '')
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	883 urllib.request.urlretrieve(url, filename=local)
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	884 if local.endswith('.txt'):
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	885 return open(local, encoding='utf-8')
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	886 else:
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	887 # Unihan.zip
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	888 return open(local, 'rb')
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	889
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	890 # --------------------------------------------------------------------
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	891 # the following support code is taken from the unidb utilities
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	892 # Copyright (c) 1999-2000 by Secret Labs AB
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	893
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	894 # load a unicode-data file from disk
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	895
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	896 class UnicodeData:
42595 f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	897 # Record structure:
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	898 # [ID, name, category, combining, bidi, decomp, (6)
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	899 # decimal, digit, numeric, bidi-mirrored, Unicode-1-name, (11)
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	900 # ISO-comment, uppercase, lowercase, titlecase, ea-width, (16)
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	901 # derived-props] (17)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	902
65217 cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	903 def __init__(self, version,
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	904 linebreakprops=False,
66392 d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	905 expand=1,
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	906 cjk_check=True):
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	907 self.changed = []
25852 1f5f6481886d Update to Unicode 3.2 database. Martin v. Löwis <martin@v.loewis.de> parents: 25219 diff changeset	908 table = [None] * 0x110000
72547 b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	909 with open_data(UNICODE_DATA, version) as file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	910 while 1:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	911 s = file.readline()
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	912 if not s:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	913 break
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	914 s = s.strip().split(";")
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	915 char = int(s[0], 16)
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	916 table[char] = s
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	917
66392 d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	918 cjk_ranges_found = []
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	919
26394 491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	920 # expand first-last ranges
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	921 if expand:
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	922 field = None
26394 491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	923 for i in range(0, 0x110000):
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	924 s = table[i]
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	925 if s:
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	926 if s[1][-6:] == "First>":
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	927 s[1] = ""
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	928 field = s
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	929 elif s[1][-5:] == "Last>":
66392 d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	930 if s[1].startswith("<CJK Ideograph"):
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	931 cjk_ranges_found.append((field[0],
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	932 s[0]))
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	933 s[1] = ""
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	934 field = None
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	935 elif field:
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	936 f2 = field[:]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	937 f2[0] = "%X" % i
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	938 table[i] = f2
66392 d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	939 if cjk_check and cjk_ranges != cjk_ranges_found:
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	940 raise ValueError("CJK ranges deviate: have %r" % cjk_ranges_found)
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	941
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	942 # public attributes
65217 cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	943 self.filename = UNICODE_DATA % ''
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	944 self.table = table
47103 63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893. Georg Brandl <georg@python.org> parents: 45049 diff changeset	945 self.chars = list(range(0x110000)) # unicode 3.2
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	946
73044 a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	947 # check for name aliases and named sequences, see #12753
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	948 # aliases and named sequences are not in 3.2.0
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	949 if version != '3.2.0':
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	950 self.aliases = []
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	951 # store aliases in the Private Use Area 15, in range U+F0000..U+F00FF,
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	952 # in order to take advantage of the compression and lookup
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	953 # algorithms used for the other characters
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	954 pua_index = NAME_ALIASES_START
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	955 with open_data(NAME_ALIASES, version) as file:
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	956 for s in file:
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	957 s = s.strip()
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	958 if not s or s.startswith('#'):
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	959 continue
75107 cd21e6d30a1a update to Unicode 6.1 Benjamin Peterson <benjamin@python.org> parents: 74423 diff changeset	960 char, name, abbrev = s.split(';')
73044 a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	961 char = int(char, 16)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	962 self.aliases.append((name, char))
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	963 # also store the name in the PUA 1
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	964 self.table[pua_index][1] = name
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	965 pua_index += 1
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	966 assert pua_index - NAME_ALIASES_START == len(self.aliases)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	967
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	968 self.named_sequences = []
85406 c57d99a8fc42 #18803: fix more typos. Patch by Févry Thibault. Ezio Melotti <ezio.melotti@gmail.com> parents: 77976 diff changeset	969 # store named sequences in the PUA 1, in range U+F0100..,
73044 a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	970 # in order to take advantage of the compression and lookup
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	971 # algorithms used for the other characters.
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	972
75107 cd21e6d30a1a update to Unicode 6.1 Benjamin Peterson <benjamin@python.org> parents: 74423 diff changeset	973 assert pua_index < NAMED_SEQUENCES_START
73044 a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	974 pua_index = NAMED_SEQUENCES_START
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	975 with open_data(NAMED_SEQUENCES, version) as file:
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	976 for s in file:
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	977 s = s.strip()
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	978 if not s or s.startswith('#'):
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	979 continue
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	980 name, chars = s.split(';')
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	981 chars = tuple(int(char, 16) for char in chars.split())
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	982 # check that the structure defined in makeunicodename is OK
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	983 assert 2 <= len(chars) <= 4, "change the Py_UCS2 array size"
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	984 assert all(c <= 0xFFFF for c in chars), ("use Py_UCS4 in "
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	985 "the NamedSequence struct and in unicodedata_lookup")
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	986 self.named_sequences.append((name, chars))
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	987 # also store these in the PUA 1
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	988 self.table[pua_index][1] = name
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	989 pua_index += 1
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	990 assert pua_index - NAMED_SEQUENCES_START == len(self.named_sequences)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	991
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	992 self.exclusions = {}
72547 b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	993 with open_data(COMPOSITION_EXCLUSIONS, version) as file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	994 for s in file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	995 s = s.strip()
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	996 if not s:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	997 continue
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	998 if s[0] == '#':
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	999 continue
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1000 char = int(s.split()[0],16)
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1001 self.exclusions[char] = 1
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	1002
32197 3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	1003 widths = [None] * 0x110000
72547 b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1004 with open_data(EASTASIAN_WIDTH, version) as file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1005 for s in file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1006 s = s.strip()
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1007 if not s:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1008 continue
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1009 if s[0] == '#':
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1010 continue
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1011 s = s.split()[0].split(';')
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1012 if '..' in s[0]:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1013 first, last = [int(c, 16) for c in s[0].split('..')]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1014 chars = list(range(first, last+1))
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1015 else:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1016 chars = [int(s[0], 16)]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1017 for char in chars:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1018 widths[char] = s[1]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1019
32197 3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	1020 for i in range(0, 0x110000):
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	1021 if table[i] is not None:
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	1022 table[i].append(widths[i])
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	1023
42595 f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	1024 for i in range(0, 0x110000):
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	1025 if table[i] is not None:
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	1026 table[i].append(set())
72547 b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1027
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1028 with open_data(DERIVED_CORE_PROPERTIES, version) as file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1029 for s in file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1030 s = s.split('#', 1)[0].strip()
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1031 if not s:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1032 continue
42595 f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	1033
72547 b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1034 r, p = s.split(";")
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1035 r = r.strip()
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1036 p = p.strip()
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1037 if ".." in r:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1038 first, last = [int(c, 16) for c in r.split('..')]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1039 chars = list(range(first, last+1))
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1040 else:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1041 chars = [int(r, 16)]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1042 for char in chars:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1043 if table[char]:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1044 # Some properties (e.g. Default_Ignorable_Code_Point)
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1045 # apply to unassigned code points; ignore them
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1046 table[char][-1].add(p)
42595 f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	1047
72547 b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1048 with open_data(LINE_BREAK, version) as file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1049 for s in file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1050 s = s.partition('#')[0]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1051 s = [i.strip() for i in s.split(';')]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1052 if len(s) < 2 or s[1] not in MANDATORY_LINE_BREAKS:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1053 continue
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1054 if '..' not in s[0]:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1055 first = last = int(s[0], 16)
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1056 else:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1057 first, last = [int(c, 16) for c in s[0].split('..')]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1058 for char in range(first, last+1):
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1059 table[char][-1].add('Line_Break')
60086 bf3c48f2417c Merged revisions 79494,79496 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 59720 diff changeset	1060
65217 cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	1061 # We only want the quickcheck properties
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	1062 # Format: NF?_QC; Y(es)/N(o)/M(aybe)
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	1063 # Yes is the default, hence only N and M occur
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	1064 # In 3.2.0, the format was different (NF?_NO)
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	1065 # The parsing will incorrectly determine these as
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	1066 # "yes", however, unicodedata.c will not perform quickchecks
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	1067 # for older versions, and no delta records will be created.
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	1068 quickchecks = [0] * 0x110000
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	1069 qc_order = 'NFD_QC NFKD_QC NFC_QC NFKC_QC'.split()
72547 b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1070 with open_data(DERIVEDNORMALIZATION_PROPS, version) as file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1071 for s in file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1072 if '#' in s:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1073 s = s[:s.index('#')]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1074 s = [i.strip() for i in s.split(';')]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1075 if len(s) < 2 or s[1] not in qc_order:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1076 continue
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1077 quickcheck = 'MN'.index(s[2]) + 1 # Maybe or No
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1078 quickcheck_shift = qc_order.index(s[1])*2
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1079 quickcheck <<= quickcheck_shift
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1080 if '..' not in s[0]:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1081 first = last = int(s[0], 16)
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1082 else:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1083 first, last = [int(c, 16) for c in s[0].split('..')]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1084 for char in range(first, last+1):
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1085 assert not (quickchecks[char]>>quickcheck_shift)&3
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1086 quickchecks[char] \|= quickcheck
65217 cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	1087 for i in range(0, 0x110000):
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	1088 if table[i] is not None:
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	1089 table[i].append(quickchecks[i])
53554 423f8f9b07b1 Merged revisions 72054 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 53402 diff changeset	1090
72547 b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1091 with open_data(UNIHAN, version) as file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1092 zip = zipfile.ZipFile(file)
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1093 if version == '3.2.0':
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1094 data = zip.open('Unihan-3.2.0.txt').read()
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1095 else:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1096 data = zip.open('Unihan_NumericValues.txt').read()
65217 cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	1097 for line in data.decode("utf-8").splitlines():
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	1098 if not line.startswith('U+'):
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	1099 continue
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	1100 code, tag, value = line.split(None, 3)[:3]
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	1101 if tag not in ('kAccountingNumeric', 'kPrimaryNumeric',
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	1102 'kOtherNumeric'):
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	1103 continue
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	1104 value = value.strip().replace(',', '')
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	1105 i = int(code[2:], 16)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	1106 # Patch the numeric field
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	1107 if table[i] is not None:
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	1108 table[i][8] = value
74338 f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	1109 sc = self.special_casing = {}
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	1110 with open_data(SPECIAL_CASING, version) as file:
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	1111 for s in file:
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	1112 s = s[:-1].split('#', 1)[0]
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	1113 if not s:
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	1114 continue
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	1115 data = s.split("; ")
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	1116 if data[4]:
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	1117 # We ignore all conditionals (since they depend on
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	1118 # languages) except for one, which is hardcoded. See
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	1119 # handle_capital_sigma in unicodeobject.c.
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	1120 continue
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	1121 c = int(data[0], 16)
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	1122 lower = [int(char, 16) for char in data[1].split()]
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	1123 title = [int(char, 16) for char in data[2].split()]
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	1124 upper = [int(char, 16) for char in data[3].split()]
f7e05d205a52 use full unicode mappings for upper/lower/title case (#12736) Benjamin Peterson <benjamin@python.org> parents: 73044 diff changeset	1125 sc[c] = (lower, title, upper)
74405 d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	1126 cf = self.case_folding = {}
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	1127 if version != '3.2.0':
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	1128 with open_data(CASE_FOLDING, version) as file:
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	1129 for s in file:
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	1130 s = s[:-1].split('#', 1)[0]
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	1131 if not s:
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	1132 continue
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	1133 data = s.split("; ")
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	1134 if data[1] in "CF":
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	1135 c = int(data[0], 16)
d4669f43d05f add str.casefold() (closes #13752) Benjamin Peterson <benjamin@python.org> parents: 74338 diff changeset	1136 cf[c] = [int(char, 16) for char in data[2].split()]
53554 423f8f9b07b1 Merged revisions 72054 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 53402 diff changeset	1137
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1138 def uselatin1(self):
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1139 # restrict character range to ISO Latin 1
47103 63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893. Georg Brandl <georg@python.org> parents: 45049 diff changeset	1140 self.chars = list(range(256))
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1141
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1142 # hash table tools
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1143
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1144 # this is a straight-forward reimplementation of Python's built-in
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1145 # dictionary type, using a static data structure, and a custom string
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1146 # hash algorithm.
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1147
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1148 def myhash(s, magic):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1149 h = 0
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	1150 for c in map(ord, s.upper()):
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1151 h = (h * magic) + c
42084 229c28cb9afb Merged revisions 55817-55961 via svnmerge from Guido van Rossum <guido@python.org> parents: 41642 diff changeset	1152 ix = h & 0xff000000
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1153 if ix:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1154 h = (h ^ ((ix>>24) & 0xff)) & 0x00ffffff
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1155 return h
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1156
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1157 SIZES = [
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1158 (4,3), (8,3), (16,3), (32,5), (64,3), (128,3), (256,29), (512,17),
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1159 (1024,9), (2048,5), (4096,83), (8192,27), (16384,43), (32768,3),
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1160 (65536,45), (131072,9), (262144,39), (524288,39), (1048576,9),
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1161 (2097152,5), (4194304,3), (8388608,33), (16777216,27)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1162 ]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1163
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1164 class Hash:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1165 def __init__(self, name, data, magic):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1166 # turn a (key, value) list into a static hash table structure
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1167
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1168 # determine table size
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1169 for size, poly in SIZES:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1170 if size > len(data):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1171 poly = size + poly
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1172 break
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1173 else:
68585 bf94b6a73fc8 #11565: Fix several typos. Patch by Piotr Kasprzyk. Ezio Melotti <ezio.melotti@gmail.com> parents: 53554 diff changeset	1174 raise AssertionError("ran out of polynomials")
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1175
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	1176 print(size, "slots in hash table")
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1177
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1178 table = [None] * size
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1179
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1180 mask = size-1
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1181
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1182 n = 0
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1183
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1184 hash = myhash
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1185
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1186 # initialize hash table
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1187 for key, value in data:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1188 h = hash(key, magic)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1189 i = (~h) & mask
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1190 v = table[i]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1191 if v is None:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1192 table[i] = value
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1193 continue
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1194 incr = (h ^ (h >> 3)) & mask;
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1195 if not incr:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1196 incr = mask
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1197 while 1:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1198 n = n + 1
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1199 i = (i + incr) & mask
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1200 v = table[i]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1201 if v is None:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1202 table[i] = value
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1203 break
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1204 incr = incr << 1
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1205 if incr > mask:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1206 incr = incr ^ poly
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1207
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	1208 print(n, "collisions")
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1209 self.collisions = n
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1210
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1211 for i in range(len(table)):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1212 if table[i] is None:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1213 table[i] = 0
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1214
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1215 self.data = Array(name + "_hash", table)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1216 self.magic = magic
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1217 self.name = name
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1218 self.size = size
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1219 self.poly = poly
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1220
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	1221 def dump(self, file, trace):
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1222 # write data to file, as a C array
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	1223 self.data.dump(file, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1224 file.write("#define %s_magic %d\n" % (self.name, self.magic))
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1225 file.write("#define %s_size %d\n" % (self.name, self.size))
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1226 file.write("#define %s_poly %d\n" % (self.name, self.poly))
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1227
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1228 # stuff to deal with arrays of unsigned integers
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1229
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1230 class Array:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1231
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1232 def __init__(self, name, data):
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1233 self.name = name
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1234 self.data = data
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1235
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	1236 def dump(self, file, trace=0):
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1237 # write data to file, as a C array
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1238 size = getsize(self.data)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	1239 if trace:
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	1240 print(self.name+":", size*len(self.data), "bytes", file=sys.stderr)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1241 file.write("static ")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1242 if size == 1:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1243 file.write("unsigned char")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1244 elif size == 2:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1245 file.write("unsigned short")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1246 else:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1247 file.write("unsigned int")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1248 file.write(" " + self.name + "[] = {\n")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1249 if self.data:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1250 s = " "
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1251 for item in self.data:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1252 i = str(item) + ", "
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1253 if len(s) + len(i) > 78:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1254 file.write(s + "\n")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1255 s = " " + i
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1256 else:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1257 s = s + i
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	1258 if s.strip():
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1259 file.write(s + "\n")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1260 file.write("};\n\n")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1261
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1262 def getsize(data):
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1263 # return smallest possible integer size for the given array
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1264 maxdata = max(data)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1265 if maxdata < 256:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1266 return 1
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1267 elif maxdata < 65536:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1268 return 2
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1269 else:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1270 return 4
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1271
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1272 def splitbins(t, trace=0):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1273 """t, trace=0 -> (t1, t2, shift). Split a table to save space.
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1274
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1275 t is a sequence of ints. This function can be useful to save space if
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1276 many of the ints are the same. t1 and t2 are lists of ints, and shift
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1277 is an int, chosen to minimize the combined size of t1 and t2 (in C
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1278 code), and where for each i in range(len(t)),
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1279 t[i] == t2[(t1[i >> shift] << shift) + (i & mask)]
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1280 where mask is a bitmask isolating the last "shift" bits.
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1281
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	1282 If optional arg trace is non-zero (default zero), progress info
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	1283 is printed to sys.stderr. The higher the value, the more info
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	1284 you'll get.
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1285 """
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1286
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1287 if trace:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1288 def dump(t1, t2, shift, bytes):
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	1289 print("%d+%d bins at shift %d; %d bytes" % (
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	1290 len(t1), len(t2), shift, bytes), file=sys.stderr)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	1291 print("Size of original table:", len(t)*getsize(t), \
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	1292 "bytes", file=sys.stderr)
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1293 n = len(t)-1 # last valid index
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1294 maxshift = 0 # the most we can shift n and still have something left
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1295 if n > 0:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1296 while n >> 1:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1297 n >>= 1
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1298 maxshift += 1
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1299 del n
44160 9d152530910a Removed PyInt_GetMax and sys.maxint Christian Heimes <christian@cheimes.de> parents: 42715 diff changeset	1300 bytes = sys.maxsize # smallest total size so far
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1301 t = tuple(t) # so slices can be dict keys
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1302 for shift in range(maxshift + 1):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1303 t1 = []
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1304 t2 = []
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1305 size = 2**shift
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1306 bincache = {}
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1307 for i in range(0, len(t), size):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1308 bin = t[i:i+size]
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1309 index = bincache.get(bin)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1310 if index is None:
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1311 index = len(t2)
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1312 bincache[bin] = index
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1313 t2.extend(bin)
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1314 t1.append(index >> shift)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1315 # determine memory size
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1316 b = len(t1)getsize(t1) + len(t2)getsize(t2)
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	1317 if trace > 1:
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1318 dump(t1, t2, shift, b)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1319 if b < bytes:
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1320 best = t1, t2, shift
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1321 bytes = b
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1322 t1, t2, shift = best
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1323 if trace:
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	1324 print("Best:", end=' ', file=sys.stderr)
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1325 dump(t1, t2, shift, bytes)
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1326 if __debug__:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1327 # exhaustively verify that the decomposition is correct
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1328 mask = ~((~0) << shift) # i.e., low-bit mask of shift bits
41642 a6eb6acfe04a Merged revisions 55007-55179 via svnmerge from Guido van Rossum <guido@python.org> parents: 36940 diff changeset	1329 for i in range(len(t)):
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1330 assert t[i] == t2[(t1[i >> shift] << shift) + (i & mask)]
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1331 return best
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1332
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1333 if __name__ == "__main__":
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	1334 maketables(1)

Mercurial > cpython-lru_cache-weakref

annotate Tools/unicode/makeunicodedata.py @ 87690:66c1c9f32567 default tip