cpython-lru_cache-weakref: Tools/unicode/makeunicodedata.py annotate

annotate Tools/unicode/makeunicodedata.py @ 73798:6cde416ef03b

Credit Per Øyvind Karlsen for the initial implementation of the lzma module (issue #6715).

author	Nadeem Vawda <nadeem.vawda@gmail.com>
date	Thu, 01 Dec 2011 01:18:27 +0200
parents	a985d733b3a3
children	f7e05d205a52

rev	line source
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1 #
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	2 # (re)generate unicode property and type databases
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	3 #
26397 b27c1cad77ea Add unidata_version. Bump generator version number. Martin v. Löwis <martin@v.loewis.de> parents: 26394 diff changeset	4 # this script converts a unicode 3.2 database file to
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	5 # Modules/unicodedata_db.h, Modules/unicodename_db.h,
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	6 # and Objects/unicodetype_db.h
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	7 #
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	8 # history:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	9 # 2000-09-24 fl created (based on bits and pieces from unidb)
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	10 # 2000-09-25 fl merged tim's splitbin fixes, separate decomposition table
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	11 # 2000-09-25 fl added character type table
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	12 # 2000-09-26 fl added LINEBREAK, DECIMAL, and DIGIT flags/fields (2.0)
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	13 # 2000-11-03 fl expand first/last ranges
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	14 # 2001-01-19 fl added character name tables (2.1)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	15 # 2001-01-21 fl added decomp compression; dynamic phrasebook threshold
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	16 # 2002-09-11 wd use string methods
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	17 # 2002-10-18 mvl update to Unicode 3.2
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	18 # 2002-10-22 mvl generate NFC tables
26394 491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	19 # 2002-11-24 mvl expand all ranges, sort names version-independently
26397 b27c1cad77ea Add unidata_version. Bump generator version number. Martin v. Löwis <martin@v.loewis.de> parents: 26394 diff changeset	20 # 2002-11-25 mvl add UNIDATA_VERSION
32197 3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	21 # 2004-05-29 perky add east asian width information
36940 e4465d2737c4 Add changelog entry. Martin v. Löwis <martin@v.loewis.de> parents: 36937 diff changeset	22 # 2006-03-10 mvl update to Unicode 4.1; add UCD 3.2 delta
48427 4e6859c6d38c Issue #3282: str.isprintable() should return False for undefined Unicode characters. Georg Brandl <georg@python.org> parents: 47680 diff changeset	23 # 2008-06-11 gb add PRINTABLE_MASK for Atsuo Ishimoto's ascii() patch
73044 a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	24 # 2011-10-21 ezio add support for name aliases and named sequences
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	25 #
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	26 # written by Fredrik Lundh (fredrik@pythonware.com)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	27 #
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	28
73044 a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	29 import os
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	30 import sys
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	31 import zipfile
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	32
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	33 from textwrap import dedent
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	34 from operator import itemgetter
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	35
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	36 SCRIPT = sys.argv[0]
64146 8f97c64b2333 #5127: Even on narrow unicode builds, the C functions that access the Unicode Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 60086 diff changeset	37 VERSION = "3.2"
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	38
26397 b27c1cad77ea Add unidata_version. Bump generator version number. Martin v. Löwis <martin@v.loewis.de> parents: 26394 diff changeset	39 # The Unicode Database
65217 cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	40 UNIDATA_VERSION = "6.0.0"
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	41 UNICODE_DATA = "UnicodeData%s.txt"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	42 COMPOSITION_EXCLUSIONS = "CompositionExclusions%s.txt"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	43 EASTASIAN_WIDTH = "EastAsianWidth%s.txt"
65217 cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	44 UNIHAN = "Unihan%s.zip"
42595 f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	45 DERIVED_CORE_PROPERTIES = "DerivedCoreProperties%s.txt"
53554 423f8f9b07b1 Merged revisions 72054 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 53402 diff changeset	46 DERIVEDNORMALIZATION_PROPS = "DerivedNormalizationProps%s.txt"
60086 bf3c48f2417c Merged revisions 79494,79496 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 59720 diff changeset	47 LINE_BREAK = "LineBreak%s.txt"
73044 a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	48 NAME_ALIASES = "NameAliases%s.txt"
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	49 NAMED_SEQUENCES = "NamedSequences%s.txt"
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	50
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	51 # Private Use Areas -- in planes 1, 15, 16
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	52 PUA_1 = range(0xE000, 0xF900)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	53 PUA_15 = range(0xF0000, 0xFFFFE)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	54 PUA_16 = range(0x100000, 0x10FFFE)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	55
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	56 # we use this ranges of PUA_15 to store name aliases and named sequences
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	57 NAME_ALIASES_START = 0xF0000
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	58 NAMED_SEQUENCES_START = 0xF0100
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	59
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	60 old_versions = ["3.2.0"]
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	61
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	62 CATEGORY_NAMES = [ "Cn", "Lu", "Ll", "Lt", "Mn", "Mc", "Me", "Nd",
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	63 "Nl", "No", "Zs", "Zl", "Zp", "Cc", "Cf", "Cs", "Co", "Cn", "Lm",
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	64 "Lo", "Pc", "Pd", "Ps", "Pe", "Pi", "Pf", "Po", "Sm", "Sc", "Sk",
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	65 "So" ]
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	66
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	67 BIDIRECTIONAL_NAMES = [ "", "L", "LRE", "LRO", "R", "AL", "RLE", "RLO",
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	68 "PDF", "EN", "ES", "ET", "AN", "CS", "NSM", "BN", "B", "S", "WS",
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	69 "ON" ]
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	70
32951 024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	71 EASTASIANWIDTH_NAMES = [ "F", "H", "W", "Na", "A", "N" ]
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	72
60086 bf3c48f2417c Merged revisions 79494,79496 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 59720 diff changeset	73 MANDATORY_LINE_BREAKS = [ "BK", "CR", "LF", "NL" ]
bf3c48f2417c Merged revisions 79494,79496 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 59720 diff changeset	74
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	75 # note: should match definitions in Objects/unicodectype.c
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	76 ALPHA_MASK = 0x01
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	77 DECIMAL_MASK = 0x02
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	78 DIGIT_MASK = 0x04
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	79 LOWER_MASK = 0x08
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	80 LINEBREAK_MASK = 0x10
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	81 SPACE_MASK = 0x20
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	82 TITLE_MASK = 0x40
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	83 UPPER_MASK = 0x80
42595 f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	84 XID_START_MASK = 0x100
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	85 XID_CONTINUE_MASK = 0x200
48427 4e6859c6d38c Issue #3282: str.isprintable() should return False for undefined Unicode characters. Georg Brandl <georg@python.org> parents: 47680 diff changeset	86 PRINTABLE_MASK = 0x400
49125 dc60293b29d5 Merged revisions 66362 via svnmerge from Martin v. Löwis <martin@v.loewis.de> parents: 48427 diff changeset	87 NODELTA_MASK = 0x800
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	88 NUMERIC_MASK = 0x1000
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	89
66392 d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	90 # these ranges need to match unicodedata.c:is_unified_ideograph
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	91 cjk_ranges = [
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	92 ('3400', '4DB5'),
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	93 ('4E00', '9FCB'),
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	94 ('20000', '2A6D6'),
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	95 ('2A700', '2B734'),
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	96 ('2B740', '2B81D')
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	97 ]
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	98
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	99 def maketables(trace=0):
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	100
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	101 print("--- Reading", UNICODE_DATA % "", "...")
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	102
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	103 version = ""
65217 cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	104 unicode = UnicodeData(UNIDATA_VERSION)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	105
47611 da47e7e135ae #2630: Implement PEP 3138. Georg Brandl <georg@python.org> parents: 47463 diff changeset	106 print(len(list(filter(None, unicode.table))), "characters")
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	107
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	108 for version in old_versions:
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	109 print("--- Reading", UNICODE_DATA % ("-"+version), "...")
66392 d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	110 old_unicode = UnicodeData(version, cjk_check=False)
47611 da47e7e135ae #2630: Implement PEP 3138. Georg Brandl <georg@python.org> parents: 47463 diff changeset	111 print(len(list(filter(None, old_unicode.table))), "characters")
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	112 merge_old_version(version, unicode, old_unicode)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	113
16091 45c065668ea5 Unicode nits: Don't include unicodedatabase.h no more. And make sure Fredrik Lundh <fredrik@pythonware.com> parents: 16090 diff changeset	114 makeunicodename(unicode, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	115 makeunicodedata(unicode, trace)
16091 45c065668ea5 Unicode nits: Don't include unicodedatabase.h no more. And make sure Fredrik Lundh <fredrik@pythonware.com> parents: 16090 diff changeset	116 makeunicodetype(unicode, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	117
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	118 # --------------------------------------------------------------------
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	119 # unicode character properties
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	120
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	121 def makeunicodedata(unicode, trace):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	122
53554 423f8f9b07b1 Merged revisions 72054 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 53402 diff changeset	123 dummy = (0, 0, 0, 0, 0, 0)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	124 table = [dummy]
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	125 cache = {0: dummy}
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	126 index = [0] * len(unicode.chars)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	127
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	128 FILE = "Modules/unicodedata_db.h"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	129
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	130 print("--- Preparing", FILE, "...")
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	131
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	132 # 1) database properties
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	133
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	134 for char in unicode.chars:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	135 record = unicode.table[char]
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	136 if record:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	137 # extract database properties
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	138 category = CATEGORY_NAMES.index(record[2])
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	139 combining = int(record[3])
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	140 bidirectional = BIDIRECTIONAL_NAMES.index(record[4])
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	141 mirrored = record[9] == "Y"
32951 024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	142 eastasianwidth = EASTASIANWIDTH_NAMES.index(record[15])
53554 423f8f9b07b1 Merged revisions 72054 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 53402 diff changeset	143 normalizationquickcheck = record[17]
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	144 item = (
53554 423f8f9b07b1 Merged revisions 72054 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 53402 diff changeset	145 category, combining, bidirectional, mirrored, eastasianwidth,
423f8f9b07b1 Merged revisions 72054 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 53402 diff changeset	146 normalizationquickcheck
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	147 )
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	148 # add entry to index and item tables
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	149 i = cache.get(item)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	150 if i is None:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	151 cache[item] = i = len(table)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	152 table.append(item)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	153 index[char] = i
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	154
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	155 # 2) decomposition data
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	156
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	157 decomp_data = [0]
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	158 decomp_prefix = [""]
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	159 decomp_index = [0] * len(unicode.chars)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	160 decomp_size = 0
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	161
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	162 comp_pairs = []
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	163 comp_first = [None] * len(unicode.chars)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	164 comp_last = [None] * len(unicode.chars)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	165
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	166 for char in unicode.chars:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	167 record = unicode.table[char]
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	168 if record:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	169 if record[5]:
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	170 decomp = record[5].split()
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	171 if len(decomp) > 19:
42715 e4ade16a694f Convert raise statements in Tools/. Collin Winter <collinw@gmail.com> parents: 42595 diff changeset	172 raise Exception("character %x has a decomposition too large for nfd_nfkd" % char)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	173 # prefix
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	174 if decomp[0][0] == "<":
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	175 prefix = decomp.pop(0)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	176 else:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	177 prefix = ""
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	178 try:
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	179 i = decomp_prefix.index(prefix)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	180 except ValueError:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	181 i = len(decomp_prefix)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	182 decomp_prefix.append(prefix)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	183 prefix = i
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	184 assert prefix < 256
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	185 # content
47103 63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893. Georg Brandl <georg@python.org> parents: 45049 diff changeset	186 decomp = [prefix + (len(decomp)<<8)] + [int(s, 16) for s in decomp]
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	187 # Collect NFC pairs
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	188 if not prefix and len(decomp) == 3 and \
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	189 char not in unicode.exclusions and \
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	190 unicode.table[decomp[1]][3] == "0":
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	191 p, l, r = decomp
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	192 comp_first[l] = 1
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	193 comp_last[r] = 1
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	194 comp_pairs.append((l,r,char))
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	195 try:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	196 i = decomp_data.index(decomp)
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	197 except ValueError:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	198 i = len(decomp_data)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	199 decomp_data.extend(decomp)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	200 decomp_size = decomp_size + len(decomp) * 2
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	201 else:
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	202 i = 0
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	203 decomp_index[char] = i
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	204
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	205 f = l = 0
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	206 comp_first_ranges = []
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	207 comp_last_ranges = []
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	208 prev_f = prev_l = None
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	209 for i in unicode.chars:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	210 if comp_first[i] is not None:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	211 comp_first[i] = f
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	212 f += 1
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	213 if prev_f is None:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	214 prev_f = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	215 elif prev_f[1]+1 == i:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	216 prev_f = prev_f[0],i
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	217 else:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	218 comp_first_ranges.append(prev_f)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	219 prev_f = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	220 if comp_last[i] is not None:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	221 comp_last[i] = l
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	222 l += 1
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	223 if prev_l is None:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	224 prev_l = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	225 elif prev_l[1]+1 == i:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	226 prev_l = prev_l[0],i
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	227 else:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	228 comp_last_ranges.append(prev_l)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	229 prev_l = (i,i)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	230 comp_first_ranges.append(prev_f)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	231 comp_last_ranges.append(prev_l)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	232 total_first = f
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	233 total_last = l
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	234
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	235 comp_data = [0](total_firsttotal_last)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	236 for f,l,char in comp_pairs:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	237 f = comp_first[f]
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	238 l = comp_last[l]
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	239 comp_data[f*total_last+l] = char
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	240
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	241 print(len(table), "unique properties")
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	242 print(len(decomp_prefix), "unique decomposition prefixes")
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	243 print(len(decomp_data), "unique decomposition entries:", end=' ')
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	244 print(decomp_size, "bytes")
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	245 print(total_first, "first characters in NFC")
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	246 print(total_last, "last characters in NFC")
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	247 print(len(comp_pairs), "NFC pairs")
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	248
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	249 print("--- Writing", FILE, "...")
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	250
15339 c2d7451b42f6 Remove bogus stdout redirection and use of sys.__stdout__; use Fred Drake <fdrake@acm.org> parents: 14856 diff changeset	251 fp = open(FILE, "w")
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	252 print("/* this file was generated by %s %s */" % (SCRIPT, VERSION), file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	253 print(file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	254 print('#define UNIDATA_VERSION "%s"' % UNIDATA_VERSION, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	255 print("/* a list of unique database records */", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	256 print("const _PyUnicode_DatabaseRecord _PyUnicode_Database_Records[] = {", file=fp)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	257 for item in table:
53554 423f8f9b07b1 Merged revisions 72054 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 53402 diff changeset	258 print(" {%d, %d, %d, %d, %d, %d}," % item, file=fp)
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	259 print("};", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	260 print(file=fp)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	261
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	262 print("/* Reindexing of NFC first characters. */", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	263 print("#define TOTAL_FIRST",total_first, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	264 print("#define TOTAL_LAST",total_last, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	265 print("struct reindex{int start;short count,index;};", file=fp)
47680 b5e02775fffc Merged revisions 64226 via svnmerge from Martin v. Löwis <martin@v.loewis.de> parents: 47611 diff changeset	266 print("static struct reindex nfc_first[] = {", file=fp)
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	267 for start,end in comp_first_ranges:
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	268 print(" { %d, %d, %d}," % (start,end-start,comp_first[start]), file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	269 print(" {0,0,0}", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	270 print("};\n", file=fp)
47680 b5e02775fffc Merged revisions 64226 via svnmerge from Martin v. Löwis <martin@v.loewis.de> parents: 47611 diff changeset	271 print("static struct reindex nfc_last[] = {", file=fp)
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	272 for start,end in comp_last_ranges:
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	273 print(" { %d, %d, %d}," % (start,end-start,comp_last[start]), file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	274 print(" {0,0,0}", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	275 print("};\n", file=fp)
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	276
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	277 # FIXME: <fl> the following tables could be made static, and
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	278 # the support code moved into unicodedatabase.c
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	279
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	280 print("/* string literals */", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	281 print("const char *_PyUnicode_CategoryNames[] = {", file=fp)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	282 for name in CATEGORY_NAMES:
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	283 print(" \"%s\"," % name, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	284 print(" NULL", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	285 print("};", file=fp)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	286
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	287 print("const char *_PyUnicode_BidirectionalNames[] = {", file=fp)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	288 for name in BIDIRECTIONAL_NAMES:
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	289 print(" \"%s\"," % name, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	290 print(" NULL", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	291 print("};", file=fp)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	292
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	293 print("const char *_PyUnicode_EastAsianWidthNames[] = {", file=fp)
32951 024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	294 for name in EASTASIANWIDTH_NAMES:
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	295 print(" \"%s\"," % name, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	296 print(" NULL", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	297 print("};", file=fp)
32951 024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32197 diff changeset	298
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	299 print("static const char *decomp_prefix[] = {", file=fp)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	300 for name in decomp_prefix:
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	301 print(" \"%s\"," % name, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	302 print(" NULL", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	303 print("};", file=fp)
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	304
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	305 # split record index table
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	306 index1, index2, shift = splitbins(index, trace)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	307
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	308 print("/* index tables for the database records */", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	309 print("#define SHIFT", shift, file=fp)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	310 Array("index1", index1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	311 Array("index2", index2).dump(fp, trace)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	312
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	313 # split decomposition index table
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	314 index1, index2, shift = splitbins(decomp_index, trace)
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14834 diff changeset	315
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	316 print("/* decomposition data */", file=fp)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	317 Array("decomp_data", decomp_data).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	318
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	319 print("/* index tables for the decomposition data */", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	320 print("#define DECOMP_SHIFT", shift, file=fp)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	321 Array("decomp_index1", index1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	322 Array("decomp_index2", index2).dump(fp, trace)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	323
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	324 index, index2, shift = splitbins(comp_data, trace)
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	325 print("/* NFC pairs */", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	326 print("#define COMP_SHIFT", shift, file=fp)
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	327 Array("comp_index", index).dump(fp, trace)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	328 Array("comp_data", index2).dump(fp, trace)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	329
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	330 # Generate delta tables for old versions
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	331 for version, table, normalization in unicode.changed:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	332 cversion = version.replace(".","_")
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	333 records = [table[0]]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	334 cache = {table[0]:0}
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	335 index = [0] * len(table)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	336 for i, record in enumerate(table):
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	337 try:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	338 index[i] = cache[record]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	339 except KeyError:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	340 index[i] = cache[record] = len(records)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	341 records.append(record)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	342 index1, index2, shift = splitbins(index, trace)
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	343 print("static const change_record change_records_%s[] = {" % cversion, file=fp)
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	344 for record in records:
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	345 print("\t{ %s }," % ", ".join(map(str,record)), file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	346 print("};", file=fp)
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	347 Array("changes_%s_index" % cversion, index1).dump(fp, trace)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	348 Array("changes_%s_data" % cversion, index2).dump(fp, trace)
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	349 print("static const change_record* get_change_%s(Py_UCS4 n)" % cversion, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	350 print("{", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	351 print("\tint index;", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	352 print("\tif (n >= 0x110000) index = 0;", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	353 print("\telse {", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	354 print("\t\tindex = changes_%s_index[n>>%d];" % (cversion, shift), file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	355 print("\t\tindex = changes_%s_data[(index<<%d)+(n & %d)];" % \
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	356 (cversion, shift, ((1<<shift)-1)), file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	357 print("\t}", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	358 print("\treturn change_records_%s+index;" % cversion, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	359 print("}\n", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	360 print("static Py_UCS4 normalization_%s(Py_UCS4 n)" % cversion, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	361 print("{", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	362 print("\tswitch(n) {", file=fp)
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	363 for k, v in normalization:
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	364 print("\tcase %s: return 0x%s;" % (hex(k), v), file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	365 print("\tdefault: return 0;", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	366 print("\t}\n}\n", file=fp)
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	367
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	368 fp.close()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	369
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	370 # --------------------------------------------------------------------
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	371 # unicode character type tables
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	372
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	373 def makeunicodetype(unicode, trace):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	374
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	375 FILE = "Objects/unicodetype_db.h"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	376
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	377 print("--- Preparing", FILE, "...")
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	378
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	379 # extract unicode types
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	380 dummy = (0, 0, 0, 0, 0, 0)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	381 table = [dummy]
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	382 cache = {0: dummy}
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	383 index = [0] * len(unicode.chars)
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	384 numeric = {}
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	385 spaces = []
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	386 linebreaks = []
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	387
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	388 for char in unicode.chars:
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	389 record = unicode.table[char]
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	390 if record:
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	391 # extract database properties
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	392 category = record[2]
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	393 bidirectional = record[4]
42595 f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	394 properties = record[16]
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	395 flags = 0
49125 dc60293b29d5 Merged revisions 66362 via svnmerge from Martin v. Löwis <martin@v.loewis.de> parents: 48427 diff changeset	396 delta = True
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	397 if category in ["Lm", "Lt", "Lu", "Ll", "Lo"]:
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	398 flags \|= ALPHA_MASK
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	399 if category == "Ll":
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	400 flags \|= LOWER_MASK
60086 bf3c48f2417c Merged revisions 79494,79496 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 59720 diff changeset	401 if 'Line_Break' in properties or bidirectional == "B":
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	402 flags \|= LINEBREAK_MASK
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	403 linebreaks.append(char)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	404 if category == "Zs" or bidirectional in ("WS", "B", "S"):
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	405 flags \|= SPACE_MASK
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	406 spaces.append(char)
14856 83aff1ea5b70 - don't set the titlecase flag for uppercase letters (sorry, tim) Fredrik Lundh <fredrik@pythonware.com> parents: 14852 diff changeset	407 if category == "Lt":
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	408 flags \|= TITLE_MASK
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	409 if category == "Lu":
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	410 flags \|= UPPER_MASK
52364 ca02bd8c96f2 Merged revisions 70610 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 51122 diff changeset	411 if char == ord(" ") or category[0] not in ("C", "Z"):
48427 4e6859c6d38c Issue #3282: str.isprintable() should return False for undefined Unicode characters. Georg Brandl <georg@python.org> parents: 47680 diff changeset	412 flags \|= PRINTABLE_MASK
42595 f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	413 if "XID_Start" in properties:
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	414 flags \|= XID_START_MASK
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	415 if "XID_Continue" in properties:
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	416 flags \|= XID_CONTINUE_MASK
49125 dc60293b29d5 Merged revisions 66362 via svnmerge from Martin v. Löwis <martin@v.loewis.de> parents: 48427 diff changeset	417 # use delta predictor for upper/lower/title if it fits
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	418 if record[12]:
53402 2f8e535f9a9b Merged revisions 71896 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 52364 diff changeset	419 upper = int(record[12], 16)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	420 else:
53402 2f8e535f9a9b Merged revisions 71896 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 52364 diff changeset	421 upper = char
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	422 if record[13]:
53402 2f8e535f9a9b Merged revisions 71896 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 52364 diff changeset	423 lower = int(record[13], 16)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	424 else:
53402 2f8e535f9a9b Merged revisions 71896 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 52364 diff changeset	425 lower = char
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	426 if record[14]:
53402 2f8e535f9a9b Merged revisions 71896 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 52364 diff changeset	427 title = int(record[14], 16)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	428 else:
53402 2f8e535f9a9b Merged revisions 71896 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 52364 diff changeset	429 # UCD.html says that a missing title char means that
2f8e535f9a9b Merged revisions 71896 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 52364 diff changeset	430 # it defaults to the uppercase character, not to the
2f8e535f9a9b Merged revisions 71896 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 52364 diff changeset	431 # character itself. Apparently, in the current UCD (5.x)
2f8e535f9a9b Merged revisions 71896 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 52364 diff changeset	432 # this feature is never used
2f8e535f9a9b Merged revisions 71896 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 52364 diff changeset	433 title = upper
2f8e535f9a9b Merged revisions 71896 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 52364 diff changeset	434 upper_d = upper - char
2f8e535f9a9b Merged revisions 71896 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 52364 diff changeset	435 lower_d = lower - char
2f8e535f9a9b Merged revisions 71896 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 52364 diff changeset	436 title_d = title - char
2f8e535f9a9b Merged revisions 71896 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 52364 diff changeset	437 if -32768 <= upper_d <= 32767 and \
2f8e535f9a9b Merged revisions 71896 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 52364 diff changeset	438 -32768 <= lower_d <= 32767 and \
2f8e535f9a9b Merged revisions 71896 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 52364 diff changeset	439 -32768 <= title_d <= 32767:
2f8e535f9a9b Merged revisions 71896 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 52364 diff changeset	440 # use deltas
2f8e535f9a9b Merged revisions 71896 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 52364 diff changeset	441 upper = upper_d & 0xffff
2f8e535f9a9b Merged revisions 71896 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 52364 diff changeset	442 lower = lower_d & 0xffff
2f8e535f9a9b Merged revisions 71896 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 52364 diff changeset	443 title = title_d & 0xffff
2f8e535f9a9b Merged revisions 71896 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 52364 diff changeset	444 else:
49125 dc60293b29d5 Merged revisions 66362 via svnmerge from Martin v. Löwis <martin@v.loewis.de> parents: 48427 diff changeset	445 flags \|= NODELTA_MASK
14852 463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	446 # decimal digit, integer digit
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	447 decimal = 0
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	448 if record[6]:
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	449 flags \|= DECIMAL_MASK
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	450 decimal = int(record[6])
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	451 digit = 0
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	452 if record[7]:
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	453 flags \|= DIGIT_MASK
463faf01685e unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14849 diff changeset	454 digit = int(record[7])
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	455 if record[8]:
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	456 flags \|= NUMERIC_MASK
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	457 numeric.setdefault(record[8], []).append(char)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	458 item = (
32197 3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	459 upper, lower, title, decimal, digit, flags
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	460 )
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	461 # add entry to index and item tables
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	462 i = cache.get(item)
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	463 if i is None:
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	464 cache[item] = i = len(table)
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	465 table.append(item)
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	466 index[char] = i
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	467
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	468 print(len(table), "unique character type entries")
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	469 print(sum(map(len, numeric.values())), "numeric code points")
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	470 print(len(spaces), "whitespace code points")
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	471 print(len(linebreaks), "linebreak code points")
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	472
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	473 print("--- Writing", FILE, "...")
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	474
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	475 fp = open(FILE, "w")
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	476 print("/* this file was generated by %s %s */" % (SCRIPT, VERSION), file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	477 print(file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	478 print("/* a list of unique character type descriptors */", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	479 print("const _PyUnicode_TypeRecord _PyUnicode_TypeRecords[] = {", file=fp)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	480 for item in table:
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	481 print(" {%d, %d, %d, %d, %d, %d}," % item, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	482 print("};", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	483 print(file=fp)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	484
92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	485 # split decomposition index table
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	486 index1, index2, shift = splitbins(index, trace)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	487
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	488 print("/* type indexes */", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	489 print("#define SHIFT", shift, file=fp)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	490 Array("index1", index1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	491 Array("index2", index2).dump(fp, trace)
14849 92cdbc0f55d5 unicode database compression, step 3: Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	492
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	493 # Generate code for _PyUnicode_ToNumeric()
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	494 numeric_items = sorted(numeric.items())
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	495 print('/* Returns the numeric value as double for Unicode characters', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	496 print(' * having this property, -1.0 otherwise.', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	497 print(' */', file=fp)
64146 8f97c64b2333 #5127: Even on narrow unicode builds, the C functions that access the Unicode Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 60086 diff changeset	498 print('double _PyUnicode_ToNumeric(Py_UCS4 ch)', file=fp)
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	499 print('{', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	500 print(' switch (ch) {', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	501 for value, codepoints in numeric_items:
56508 e8e103488fb3 Merged revisions 75396 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 56408 diff changeset	502 # Turn text into float literals
e8e103488fb3 Merged revisions 75396 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 56408 diff changeset	503 parts = value.split('/')
e8e103488fb3 Merged revisions 75396 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 56408 diff changeset	504 parts = [repr(float(part)) for part in parts]
e8e103488fb3 Merged revisions 75396 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 56408 diff changeset	505 value = '/'.join(parts)
e8e103488fb3 Merged revisions 75396 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 56408 diff changeset	506
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	507 codepoints.sort()
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	508 for codepoint in codepoints:
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	509 print(' case 0x%04X:' % (codepoint,), file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	510 print(' return (double) %s;' % (value,), file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	511 print(' }', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	512 print(' return -1.0;', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	513 print('}', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	514 print(file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	515
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	516 # Generate code for _PyUnicode_IsWhitespace()
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	517 print("/* Returns 1 for Unicode characters having the bidirectional", file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	518 print(" * type 'WS', 'B' or 'S' or the category 'Zs', 0 otherwise.", file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	519 print(" */", file=fp)
64146 8f97c64b2333 #5127: Even on narrow unicode builds, the C functions that access the Unicode Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 60086 diff changeset	520 print('int _PyUnicode_IsWhitespace(register const Py_UCS4 ch)', file=fp)
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	521 print('{', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	522 print(' switch (ch) {', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	523
59720 8a28be45c242 Merged revisions 78982,78986 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 59719 diff changeset	524 for codepoint in sorted(spaces):
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	525 print(' case 0x%04X:' % (codepoint,), file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	526 print(' return 1;', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	527
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	528 print(' }', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	529 print(' return 0;', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	530 print('}', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	531 print(file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	532
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	533 # Generate code for _PyUnicode_IsLinebreak()
60086 bf3c48f2417c Merged revisions 79494,79496 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 59720 diff changeset	534 print("/* Returns 1 for Unicode characters having the line break", file=fp)
bf3c48f2417c Merged revisions 79494,79496 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 59720 diff changeset	535 print(" * property 'BK', 'CR', 'LF' or 'NL' or having bidirectional", file=fp)
bf3c48f2417c Merged revisions 79494,79496 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 59720 diff changeset	536 print(" * type 'B', 0 otherwise.", file=fp)
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	537 print(" */", file=fp)
64146 8f97c64b2333 #5127: Even on narrow unicode builds, the C functions that access the Unicode Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 60086 diff changeset	538 print('int _PyUnicode_IsLinebreak(register const Py_UCS4 ch)', file=fp)
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	539 print('{', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	540 print(' switch (ch) {', file=fp)
59720 8a28be45c242 Merged revisions 78982,78986 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 59719 diff changeset	541 for codepoint in sorted(linebreaks):
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	542 print(' case 0x%04X:' % (codepoint,), file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	543 print(' return 1;', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	544
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	545 print(' }', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	546 print(' return 0;', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	547 print('}', file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	548 print(file=fp)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	549
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	550 fp.close()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	551
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	552 # --------------------------------------------------------------------
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	553 # unicode name database
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	554
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	555 def makeunicodename(unicode, trace):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	556
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	557 FILE = "Modules/unicodename_db.h"
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	558
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	559 print("--- Preparing", FILE, "...")
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	560
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	561 # collect names
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	562 names = [None] * len(unicode.chars)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	563
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	564 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	565 record = unicode.table[char]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	566 if record:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	567 name = record[1].strip()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	568 if name and name[0] != "<":
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	569 names[char] = name + chr(0)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	570
47611 da47e7e135ae #2630: Implement PEP 3138. Georg Brandl <georg@python.org> parents: 47463 diff changeset	571 print(len(list(n for n in names if n is not None)), "distinct names")
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	572
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	573 # collect unique words from names (note that we differ between
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	574 # words inside a sentence, and words ending a sentence. the
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	575 # latter includes the trailing null byte.
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	576
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	577 words = {}
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	578 n = b = 0
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	579 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	580 name = names[char]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	581 if name:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	582 w = name.split()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	583 b = b + len(name)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	584 n = n + len(w)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	585 for w in w:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	586 l = words.get(w)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	587 if l:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	588 l.append(None)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	589 else:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	590 words[w] = [len(words)]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	591
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	592 print(n, "words in text;", b, "bytes")
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	593
42595 f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	594 wordlist = list(words.items())
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	595
26394 491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	596 # sort on falling frequency, then by name
51122 adbd77a90bde Merged revisions 69025 via svnmerge from Mark Dickinson <dickinsm@gmail.com> parents: 49125 diff changeset	597 def word_key(a):
42595 f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	598 aword, alist = a
51122 adbd77a90bde Merged revisions 69025 via svnmerge from Mark Dickinson <dickinsm@gmail.com> parents: 49125 diff changeset	599 return -len(alist), aword
adbd77a90bde Merged revisions 69025 via svnmerge from Mark Dickinson <dickinsm@gmail.com> parents: 49125 diff changeset	600 wordlist.sort(key=word_key)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	601
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	602 # figure out how many phrasebook escapes we need
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	603 escapes = 0
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	604 while escapes * 256 < len(wordlist):
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	605 escapes = escapes + 1
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	606 print(escapes, "escapes")
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	607
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	608 short = 256 - escapes
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	609
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	610 assert short > 0
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	611
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	612 print(short, "short indexes in lexicon")
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	613
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	614 # statistics
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	615 n = 0
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	616 for i in range(short):
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	617 n = n + len(wordlist[i][1])
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	618 print(n, "short indexes in phrasebook")
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	619
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	620 # pick the most commonly used words, and sort the rest on falling
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	621 # length (to maximize overlap)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	622
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	623 wordlist, wordtail = wordlist[:short], wordlist[short:]
45049 90e86ee3684b Convert some custom sort comparison functions to equivalent key functions. Raymond Hettinger <python@rcn.com> parents: 44160 diff changeset	624 wordtail.sort(key=lambda a: a[0], reverse=True)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	625 wordlist.extend(wordtail)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	626
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	627 # generate lexicon from words
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	628
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	629 lexicon_offset = [0]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	630 lexicon = ""
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	631 words = {}
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	632
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	633 # build a lexicon string
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	634 offset = 0
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	635 for w, x in wordlist:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	636 # encoding: bit 7 indicates last character in word (chr(128)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	637 # indicates the last character in an entire string)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	638 ww = w[:-1] + chr(ord(w[-1])+128)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	639 # reuse string tails, when possible
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	640 o = lexicon.find(ww)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	641 if o < 0:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	642 o = offset
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	643 lexicon = lexicon + ww
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	644 offset = offset + len(w)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	645 words[w] = len(lexicon_offset)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	646 lexicon_offset.append(o)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	647
42595 f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	648 lexicon = list(map(ord, lexicon))
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	649
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	650 # generate phrasebook from names and lexicon
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	651 phrasebook = [0]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	652 phrasebook_offset = [0] * len(unicode.chars)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	653 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	654 name = names[char]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	655 if name:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	656 w = name.split()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	657 phrasebook_offset[char] = len(phrasebook)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	658 for w in w:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	659 i = words[w]
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	660 if i < short:
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	661 phrasebook.append(i)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	662 else:
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	663 # store as two bytes
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	664 phrasebook.append((i>>8) + short)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	665 phrasebook.append(i&255)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	666
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	667 assert getsize(phrasebook) == 1
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	668
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	669 #
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	670 # unicode name hash table
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	671
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	672 # extract names
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	673 data = []
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	674 for char in unicode.chars:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	675 record = unicode.table[char]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	676 if record:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	677 name = record[1].strip()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	678 if name and name[0] != "<":
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	679 data.append((name, char))
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	680
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	681 # the magic number 47 was chosen to minimize the number of
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	682 # collisions on the current data set. if you like, change it
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	683 # and see what happens...
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	684
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	685 codehash = Hash("code", data, 47)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	686
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	687 print("--- Writing", FILE, "...")
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	688
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	689 fp = open(FILE, "w")
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	690 print("/* this file was generated by %s %s */" % (SCRIPT, VERSION), file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	691 print(file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	692 print("#define NAME_MAXLEN", 256, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	693 print(file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	694 print("/* lexicon */", file=fp)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	695 Array("lexicon", lexicon).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	696 Array("lexicon_offset", lexicon_offset).dump(fp, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	697
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	698 # split decomposition index table
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	699 offset1, offset2, shift = splitbins(phrasebook_offset, trace)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	700
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	701 print("/* code->name phrasebook */", file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	702 print("#define phrasebook_shift", shift, file=fp)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	703 print("#define phrasebook_short", short, file=fp)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	704
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	705 Array("phrasebook", phrasebook).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	706 Array("phrasebook_offset1", offset1).dump(fp, trace)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	707 Array("phrasebook_offset2", offset2).dump(fp, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	708
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	709 print("/* name->code dictionary */", file=fp)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	710 codehash.dump(fp, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	711
73044 a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	712 print(file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	713 print('static const unsigned int aliases_start = %#x;' %
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	714 NAME_ALIASES_START, file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	715 print('static const unsigned int aliases_end = %#x;' %
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	716 (NAME_ALIASES_START + len(unicode.aliases)), file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	717
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	718 print('static const unsigned int name_aliases[] = {', file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	719 for name, codepoint in unicode.aliases:
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	720 print(' 0x%04X,' % codepoint, file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	721 print('};', file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	722
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	723 # In Unicode 6.0.0, the sequences contain at most 4 BMP chars,
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	724 # so we are using Py_UCS2 seq[4]. This needs to be updated if longer
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	725 # sequences or sequences with non-BMP chars are added.
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	726 # unicodedata_lookup should be adapted too.
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	727 print(dedent("""
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	728 typedef struct NamedSequence {
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	729 int seqlen;
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	730 Py_UCS2 seq[4];
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	731 } named_sequence;
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	732 """), file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	733
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	734 print('static const unsigned int named_sequences_start = %#x;' %
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	735 NAMED_SEQUENCES_START, file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	736 print('static const unsigned int named_sequences_end = %#x;' %
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	737 (NAMED_SEQUENCES_START + len(unicode.named_sequences)), file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	738
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	739 print('static const named_sequence named_sequences[] = {', file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	740 for name, sequence in unicode.named_sequences:
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	741 seq_str = ', '.join('0x%04X' % cp for cp in sequence)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	742 print(' {%d, {%s}},' % (len(sequence), seq_str), file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	743 print('};', file=fp)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	744
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	745 fp.close()
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	746
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	747
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	748 def merge_old_version(version, new, old):
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	749 # Changes to exclusion file not implemented yet
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	750 if old.exclusions != new.exclusions:
42715 e4ade16a694f Convert raise statements in Tools/. Collin Winter <collinw@gmail.com> parents: 42595 diff changeset	751 raise NotImplementedError("exclusions differ")
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	752
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	753 # In these change records, 0xFF means "no change"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	754 bidir_changes = [0xFF]*0x110000
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	755 category_changes = [0xFF]*0x110000
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	756 decimal_changes = [0xFF]*0x110000
49125 dc60293b29d5 Merged revisions 66362 via svnmerge from Martin v. Löwis <martin@v.loewis.de> parents: 48427 diff changeset	757 mirrored_changes = [0xFF]*0x110000
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	758 # In numeric data, 0 means "no change",
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	759 # -1 means "did not have a numeric value
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	760 numeric_changes = [0] * 0x110000
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	761 # normalization_changes is a list of key-value pairs
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	762 normalization_changes = []
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	763 for i in range(0x110000):
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	764 if new.table[i] is None:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	765 # Characters unassigned in the new version ought to
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	766 # be unassigned in the old one
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	767 assert old.table[i] is None
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	768 continue
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	769 # check characters unassigned in the old version
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	770 if old.table[i] is None:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	771 # category 0 is "unassigned"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	772 category_changes[i] = 0
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	773 continue
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	774 # check characters that differ
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	775 if old.table[i] != new.table[i]:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	776 for k in range(len(old.table[i])):
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	777 if old.table[i][k] != new.table[i][k]:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	778 value = old.table[i][k]
73044 a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	779 if k == 1 and i in PUA_15:
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	780 # the name is not set in the old.table, but in the
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	781 # new.table we are using it for aliases and named seq
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	782 assert value == ''
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	783 elif k == 2:
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	784 #print "CATEGORY",hex(i), old.table[i][k], new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	785 category_changes[i] = CATEGORY_NAMES.index(value)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	786 elif k == 4:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	787 #print "BIDIR",hex(i), old.table[i][k], new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	788 bidir_changes[i] = BIDIRECTIONAL_NAMES.index(value)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	789 elif k == 5:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	790 #print "DECOMP",hex(i), old.table[i][k], new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	791 # We assume that all normalization changes are in 1:1 mappings
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	792 assert " " not in value
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	793 normalization_changes.append((i, value))
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	794 elif k == 6:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	795 #print "DECIMAL",hex(i), old.table[i][k], new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	796 # we only support changes where the old value is a single digit
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	797 assert value in "0123456789"
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	798 decimal_changes[i] = int(value)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	799 elif k == 8:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	800 # print "NUMERIC",hex(i), `old.table[i][k]`, new.table[i][k]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	801 # Since 0 encodes "no change", the old value is better not 0
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	802 if not value:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	803 numeric_changes[i] = -1
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	804 else:
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	805 numeric_changes[i] = float(value)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	806 assert numeric_changes[i] not in (0, -1)
49125 dc60293b29d5 Merged revisions 66362 via svnmerge from Martin v. Löwis <martin@v.loewis.de> parents: 48427 diff changeset	807 elif k == 9:
dc60293b29d5 Merged revisions 66362 via svnmerge from Martin v. Löwis <martin@v.loewis.de> parents: 48427 diff changeset	808 if value == 'Y':
dc60293b29d5 Merged revisions 66362 via svnmerge from Martin v. Löwis <martin@v.loewis.de> parents: 48427 diff changeset	809 mirrored_changes[i] = '1'
dc60293b29d5 Merged revisions 66362 via svnmerge from Martin v. Löwis <martin@v.loewis.de> parents: 48427 diff changeset	810 else:
dc60293b29d5 Merged revisions 66362 via svnmerge from Martin v. Löwis <martin@v.loewis.de> parents: 48427 diff changeset	811 mirrored_changes[i] = '0'
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	812 elif k == 11:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	813 # change to ISO comment, ignore
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	814 pass
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	815 elif k == 12:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	816 # change to simple uppercase mapping; ignore
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	817 pass
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	818 elif k == 13:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	819 # change to simple lowercase mapping; ignore
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	820 pass
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	821 elif k == 14:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	822 # change to simple titlecase mapping; ignore
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	823 pass
42595 f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	824 elif k == 16:
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	825 # derived property changes; not yet
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	826 pass
65217 cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	827 elif k == 17:
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	828 # normalization quickchecks are not performed
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	829 # for older versions
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	830 pass
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	831 else:
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	832 class Difference(Exception):pass
42715 e4ade16a694f Convert raise statements in Tools/. Collin Winter <collinw@gmail.com> parents: 42595 diff changeset	833 raise Difference(hex(i), k, old.table[i], new.table[i])
42595 f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	834 new.changed.append((version, list(zip(bidir_changes, category_changes,
49125 dc60293b29d5 Merged revisions 66362 via svnmerge from Martin v. Löwis <martin@v.loewis.de> parents: 48427 diff changeset	835 decimal_changes, mirrored_changes,
dc60293b29d5 Merged revisions 66362 via svnmerge from Martin v. Löwis <martin@v.loewis.de> parents: 48427 diff changeset	836 numeric_changes)),
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	837 normalization_changes))
36937 df43c04d2554 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 36924 diff changeset	838
65217 cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	839 def open_data(template, version):
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	840 local = template % ('-'+version,)
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	841 if not os.path.exists(local):
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	842 import urllib.request
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	843 if version == '3.2.0':
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	844 # irregular url structure
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	845 url = 'http://www.unicode.org/Public/3.2-Update/' + local
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	846 else:
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	847 url = ('http://www.unicode.org/Public/%s/ucd/'+template) % (version, '')
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	848 urllib.request.urlretrieve(url, filename=local)
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	849 if local.endswith('.txt'):
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	850 return open(local, encoding='utf-8')
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	851 else:
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	852 # Unihan.zip
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	853 return open(local, 'rb')
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	854
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	855 # --------------------------------------------------------------------
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	856 # the following support code is taken from the unidb utilities
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	857 # Copyright (c) 1999-2000 by Secret Labs AB
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	858
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	859 # load a unicode-data file from disk
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	860
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	861 class UnicodeData:
42595 f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	862 # Record structure:
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	863 # [ID, name, category, combining, bidi, decomp, (6)
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	864 # decimal, digit, numeric, bidi-mirrored, Unicode-1-name, (11)
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	865 # ISO-comment, uppercase, lowercase, titlecase, ea-width, (16)
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	866 # derived-props] (17)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	867
65217 cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	868 def __init__(self, version,
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	869 linebreakprops=False,
66392 d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	870 expand=1,
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	871 cjk_check=True):
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	872 self.changed = []
25852 1f5f6481886d Update to Unicode 3.2 database. Martin v. Löwis <martin@v.loewis.de> parents: 25219 diff changeset	873 table = [None] * 0x110000
72547 b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	874 with open_data(UNICODE_DATA, version) as file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	875 while 1:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	876 s = file.readline()
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	877 if not s:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	878 break
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	879 s = s.strip().split(";")
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	880 char = int(s[0], 16)
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	881 table[char] = s
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	882
66392 d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	883 cjk_ranges_found = []
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	884
26394 491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	885 # expand first-last ranges
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	886 if expand:
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	887 field = None
26394 491bfc4a0f5b Sort names independent of the Python version. Fix hex constant warning. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	888 for i in range(0, 0x110000):
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	889 s = table[i]
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	890 if s:
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	891 if s[1][-6:] == "First>":
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	892 s[1] = ""
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	893 field = s
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	894 elif s[1][-5:] == "Last>":
66392 d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	895 if s[1].startswith("<CJK Ideograph"):
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	896 cjk_ranges_found.append((field[0],
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	897 s[0]))
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	898 s[1] = ""
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	899 field = None
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	900 elif field:
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	901 f2 = field[:]
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	902 f2[0] = "%X" % i
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 32951 diff changeset	903 table[i] = f2
66392 d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	904 if cjk_check and cjk_ranges != cjk_ranges_found:
d9774121a6df Issue #10459: Update CJK character names to Unicode 6.0. Martin v. Löwis <martin@v.loewis.de> parents: 65217 diff changeset	905 raise ValueError("CJK ranges deviate: have %r" % cjk_ranges_found)
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	906
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	907 # public attributes
65217 cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	908 self.filename = UNICODE_DATA % ''
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	909 self.table = table
47103 63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893. Georg Brandl <georg@python.org> parents: 45049 diff changeset	910 self.chars = list(range(0x110000)) # unicode 3.2
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	911
73044 a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	912 # check for name aliases and named sequences, see #12753
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	913 # aliases and named sequences are not in 3.2.0
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	914 if version != '3.2.0':
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	915 self.aliases = []
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	916 # store aliases in the Private Use Area 15, in range U+F0000..U+F00FF,
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	917 # in order to take advantage of the compression and lookup
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	918 # algorithms used for the other characters
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	919 pua_index = NAME_ALIASES_START
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	920 with open_data(NAME_ALIASES, version) as file:
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	921 for s in file:
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	922 s = s.strip()
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	923 if not s or s.startswith('#'):
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	924 continue
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	925 char, name = s.split(';')
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	926 char = int(char, 16)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	927 self.aliases.append((name, char))
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	928 # also store the name in the PUA 1
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	929 self.table[pua_index][1] = name
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	930 pua_index += 1
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	931 assert pua_index - NAME_ALIASES_START == len(self.aliases)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	932
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	933 self.named_sequences = []
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	934 # store named seqences in the PUA 1, in range U+F0100..,
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	935 # in order to take advantage of the compression and lookup
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	936 # algorithms used for the other characters.
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	937
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	938 pua_index = NAMED_SEQUENCES_START
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	939 with open_data(NAMED_SEQUENCES, version) as file:
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	940 for s in file:
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	941 s = s.strip()
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	942 if not s or s.startswith('#'):
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	943 continue
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	944 name, chars = s.split(';')
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	945 chars = tuple(int(char, 16) for char in chars.split())
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	946 # check that the structure defined in makeunicodename is OK
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	947 assert 2 <= len(chars) <= 4, "change the Py_UCS2 array size"
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	948 assert all(c <= 0xFFFF for c in chars), ("use Py_UCS4 in "
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	949 "the NamedSequence struct and in unicodedata_lookup")
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	950 self.named_sequences.append((name, chars))
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	951 # also store these in the PUA 1
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	952 self.table[pua_index][1] = name
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	953 pua_index += 1
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	954 assert pua_index - NAMED_SEQUENCES_START == len(self.named_sequences)
a985d733b3a3 #12753: Add support for Unicode name aliases and named sequences. Ezio Melotti <ezio.melotti@gmail.com> parents: 72547 diff changeset	955
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	956 self.exclusions = {}
72547 b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	957 with open_data(COMPOSITION_EXCLUSIONS, version) as file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	958 for s in file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	959 s = s.strip()
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	960 if not s:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	961 continue
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	962 if s[0] == '#':
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	963 continue
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	964 char = int(s.split()[0],16)
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	965 self.exclusions[char] = 1
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 25860 diff changeset	966
32197 3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	967 widths = [None] * 0x110000
72547 b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	968 with open_data(EASTASIAN_WIDTH, version) as file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	969 for s in file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	970 s = s.strip()
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	971 if not s:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	972 continue
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	973 if s[0] == '#':
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	974 continue
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	975 s = s.split()[0].split(';')
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	976 if '..' in s[0]:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	977 first, last = [int(c, 16) for c in s[0].split('..')]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	978 chars = list(range(first, last+1))
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	979 else:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	980 chars = [int(s[0], 16)]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	981 for char in chars:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	982 widths[char] = s[1]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	983
32197 3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	984 for i in range(0, 0x110000):
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	985 if table[i] is not None:
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	986 table[i].append(widths[i])
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and Hye-Shik Chang <hyeshik@gmail.com> parents: 26397 diff changeset	987
42595 f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	988 for i in range(0, 0x110000):
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	989 if table[i] is not None:
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	990 table[i].append(set())
72547 b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	991
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	992 with open_data(DERIVED_CORE_PROPERTIES, version) as file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	993 for s in file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	994 s = s.split('#', 1)[0].strip()
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	995 if not s:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	996 continue
42595 f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	997
72547 b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	998 r, p = s.split(";")
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	999 r = r.strip()
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1000 p = p.strip()
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1001 if ".." in r:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1002 first, last = [int(c, 16) for c in r.split('..')]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1003 chars = list(range(first, last+1))
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1004 else:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1005 chars = [int(r, 16)]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1006 for char in chars:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1007 if table[char]:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1008 # Some properties (e.g. Default_Ignorable_Code_Point)
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1009 # apply to unassigned code points; ignore them
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1010 table[char][-1].add(p)
42595 f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype. Martin v. Löwis <martin@v.loewis.de> parents: 42427 diff changeset	1011
72547 b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1012 with open_data(LINE_BREAK, version) as file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1013 for s in file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1014 s = s.partition('#')[0]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1015 s = [i.strip() for i in s.split(';')]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1016 if len(s) < 2 or s[1] not in MANDATORY_LINE_BREAKS:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1017 continue
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1018 if '..' not in s[0]:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1019 first = last = int(s[0], 16)
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1020 else:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1021 first, last = [int(c, 16) for c in s[0].split('..')]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1022 for char in range(first, last+1):
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1023 table[char][-1].add('Line_Break')
60086 bf3c48f2417c Merged revisions 79494,79496 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 59720 diff changeset	1024
65217 cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	1025 # We only want the quickcheck properties
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	1026 # Format: NF?_QC; Y(es)/N(o)/M(aybe)
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	1027 # Yes is the default, hence only N and M occur
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	1028 # In 3.2.0, the format was different (NF?_NO)
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	1029 # The parsing will incorrectly determine these as
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	1030 # "yes", however, unicodedata.c will not perform quickchecks
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	1031 # for older versions, and no delta records will be created.
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	1032 quickchecks = [0] * 0x110000
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	1033 qc_order = 'NFD_QC NFKD_QC NFC_QC NFKC_QC'.split()
72547 b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1034 with open_data(DERIVEDNORMALIZATION_PROPS, version) as file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1035 for s in file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1036 if '#' in s:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1037 s = s[:s.index('#')]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1038 s = [i.strip() for i in s.split(';')]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1039 if len(s) < 2 or s[1] not in qc_order:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1040 continue
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1041 quickcheck = 'MN'.index(s[2]) + 1 # Maybe or No
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1042 quickcheck_shift = qc_order.index(s[1])*2
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1043 quickcheck <<= quickcheck_shift
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1044 if '..' not in s[0]:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1045 first = last = int(s[0], 16)
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1046 else:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1047 first, last = [int(c, 16) for c in s[0].split('..')]
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1048 for char in range(first, last+1):
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1049 assert not (quickchecks[char]>>quickcheck_shift)&3
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1050 quickchecks[char] \|= quickcheck
65217 cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	1051 for i in range(0, 0x110000):
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	1052 if table[i] is not None:
cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	1053 table[i].append(quickchecks[i])
53554 423f8f9b07b1 Merged revisions 72054 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 53402 diff changeset	1054
72547 b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1055 with open_data(UNIHAN, version) as file:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1056 zip = zipfile.ZipFile(file)
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1057 if version == '3.2.0':
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1058 data = zip.open('Unihan-3.2.0.txt').read()
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1059 else:
b0ef6a5a6dcc Fix ResourceWarnings in makeunicodedata.py. Ezio Melotti <ezio.melotti@gmail.com> parents: 68586 diff changeset	1060 data = zip.open('Unihan_NumericValues.txt').read()
65217 cbde81ef4af7 Upgrade to Unicode 6.0.0. Martin v. Löwis <martin@v.loewis.de> parents: 64687 diff changeset	1061 for line in data.decode("utf-8").splitlines():
56408 6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	1062 if not line.startswith('U+'):
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	1063 continue
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	1064 code, tag, value = line.split(None, 3)[:3]
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	1065 if tag not in ('kAccountingNumeric', 'kPrimaryNumeric',
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	1066 'kOtherNumeric'):
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	1067 continue
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	1068 value = value.strip().replace(',', '')
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	1069 i = int(code[2:], 16)
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	1070 # Patch the numeric field
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	1071 if table[i] is not None:
6c91d1803182 Merged revisions 75272-75273 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53554 diff changeset	1072 table[i][8] = value
53554 423f8f9b07b1 Merged revisions 72054 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 53402 diff changeset	1073
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1074 def uselatin1(self):
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1075 # restrict character range to ISO Latin 1
47103 63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893. Georg Brandl <georg@python.org> parents: 45049 diff changeset	1076 self.chars = list(range(256))
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1077
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1078 # hash table tools
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1079
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1080 # this is a straight-forward reimplementation of Python's built-in
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1081 # dictionary type, using a static data structure, and a custom string
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1082 # hash algorithm.
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1083
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1084 def myhash(s, magic):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1085 h = 0
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	1086 for c in map(ord, s.upper()):
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1087 h = (h * magic) + c
42084 229c28cb9afb Merged revisions 55817-55961 via svnmerge from Guido van Rossum <guido@python.org> parents: 41642 diff changeset	1088 ix = h & 0xff000000
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1089 if ix:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1090 h = (h ^ ((ix>>24) & 0xff)) & 0x00ffffff
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1091 return h
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1092
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1093 SIZES = [
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1094 (4,3), (8,3), (16,3), (32,5), (64,3), (128,3), (256,29), (512,17),
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1095 (1024,9), (2048,5), (4096,83), (8192,27), (16384,43), (32768,3),
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1096 (65536,45), (131072,9), (262144,39), (524288,39), (1048576,9),
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1097 (2097152,5), (4194304,3), (8388608,33), (16777216,27)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1098 ]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1099
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1100 class Hash:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1101 def __init__(self, name, data, magic):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1102 # turn a (key, value) list into a static hash table structure
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1103
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1104 # determine table size
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1105 for size, poly in SIZES:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1106 if size > len(data):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1107 poly = size + poly
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1108 break
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1109 else:
68585 bf94b6a73fc8 #11565: Fix several typos. Patch by Piotr Kasprzyk. Ezio Melotti <ezio.melotti@gmail.com> parents: 53554 diff changeset	1110 raise AssertionError("ran out of polynomials")
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1111
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	1112 print(size, "slots in hash table")
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1113
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1114 table = [None] * size
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1115
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1116 mask = size-1
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1117
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1118 n = 0
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1119
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1120 hash = myhash
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1121
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1122 # initialize hash table
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1123 for key, value in data:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1124 h = hash(key, magic)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1125 i = (~h) & mask
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1126 v = table[i]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1127 if v is None:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1128 table[i] = value
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1129 continue
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1130 incr = (h ^ (h >> 3)) & mask;
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1131 if not incr:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1132 incr = mask
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1133 while 1:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1134 n = n + 1
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1135 i = (i + incr) & mask
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1136 v = table[i]
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1137 if v is None:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1138 table[i] = value
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1139 break
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1140 incr = incr << 1
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1141 if incr > mask:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1142 incr = incr ^ poly
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1143
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	1144 print(n, "collisions")
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1145 self.collisions = n
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1146
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1147 for i in range(len(table)):
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1148 if table[i] is None:
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1149 table[i] = 0
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1150
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1151 self.data = Array(name + "_hash", table)
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1152 self.magic = magic
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1153 self.name = name
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1154 self.size = size
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1155 self.poly = poly
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1156
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	1157 def dump(self, file, trace):
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1158 # write data to file, as a C array
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	1159 self.data.dump(file, trace)
16080 93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1160 file.write("#define %s_magic %d\n" % (self.name, self.magic))
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1161 file.write("#define %s_size %d\n" % (self.name, self.size))
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1162 file.write("#define %s_poly %d\n" % (self.name, self.poly))
93f836432c2b forgot to check in the new makeunicodedata.py script Fredrik Lundh <fredrik@pythonware.com> parents: 15376 diff changeset	1163
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1164 # stuff to deal with arrays of unsigned integers
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1165
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1166 class Array:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1167
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1168 def __init__(self, name, data):
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1169 self.name = name
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1170 self.data = data
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1171
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	1172 def dump(self, file, trace=0):
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1173 # write data to file, as a C array
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1174 size = getsize(self.data)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 16080 diff changeset	1175 if trace:
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	1176 print(self.name+":", size*len(self.data), "bytes", file=sys.stderr)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1177 file.write("static ")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1178 if size == 1:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1179 file.write("unsigned char")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1180 elif size == 2:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1181 file.write("unsigned short")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1182 else:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1183 file.write("unsigned int")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1184 file.write(" " + self.name + "[] = {\n")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1185 if self.data:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1186 s = " "
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1187 for item in self.data:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1188 i = str(item) + ", "
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1189 if len(s) + len(i) > 78:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1190 file.write(s + "\n")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1191 s = " " + i
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1192 else:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1193 s = s + i
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 16091 diff changeset	1194 if s.strip():
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1195 file.write(s + "\n")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1196 file.write("};\n\n")
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1197
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1198 def getsize(data):
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1199 # return smallest possible integer size for the given array
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1200 maxdata = max(data)
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1201 if maxdata < 256:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1202 return 1
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1203 elif maxdata < 65536:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1204 return 2
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1205 else:
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1206 return 4
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1207
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1208 def splitbins(t, trace=0):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1209 """t, trace=0 -> (t1, t2, shift). Split a table to save space.
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1210
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1211 t is a sequence of ints. This function can be useful to save space if
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1212 many of the ints are the same. t1 and t2 are lists of ints, and shift
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1213 is an int, chosen to minimize the combined size of t1 and t2 (in C
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1214 code), and where for each i in range(len(t)),
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1215 t[i] == t2[(t1[i >> shift] << shift) + (i & mask)]
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1216 where mask is a bitmask isolating the last "shift" bits.
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1217
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	1218 If optional arg trace is non-zero (default zero), progress info
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	1219 is printed to sys.stderr. The higher the value, the more info
493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	1220 you'll get.
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1221 """
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1222
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1223 if trace:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1224 def dump(t1, t2, shift, bytes):
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	1225 print("%d+%d bins at shift %d; %d bytes" % (
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	1226 len(t1), len(t2), shift, bytes), file=sys.stderr)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	1227 print("Size of original table:", len(t)*getsize(t), \
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	1228 "bytes", file=sys.stderr)
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1229 n = len(t)-1 # last valid index
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1230 maxshift = 0 # the most we can shift n and still have something left
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1231 if n > 0:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1232 while n >> 1:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1233 n >>= 1
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1234 maxshift += 1
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1235 del n
44160 9d152530910a Removed PyInt_GetMax and sys.maxint Christian Heimes <christian@cheimes.de> parents: 42715 diff changeset	1236 bytes = sys.maxsize # smallest total size so far
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1237 t = tuple(t) # so slices can be dict keys
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1238 for shift in range(maxshift + 1):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1239 t1 = []
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1240 t2 = []
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1241 size = 2**shift
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1242 bincache = {}
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1243 for i in range(0, len(t), size):
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1244 bin = t[i:i+size]
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1245 index = bincache.get(bin)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1246 if index is None:
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1247 index = len(t2)
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1248 bincache[bin] = index
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1249 t2.extend(bin)
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1250 t1.append(index >> shift)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1251 # determine memory size
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1252 b = len(t1)getsize(t1) + len(t2)getsize(t2)
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	1253 if trace > 1:
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1254 dump(t1, t2, shift, b)
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1255 if b < bytes:
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1256 best = t1, t2, shift
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1257 bytes = b
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1258 t1, t2, shift = best
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1259 if trace:
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	1260 print("Best:", end=' ', file=sys.stderr)
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1261 dump(t1, t2, shift, bytes)
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1262 if __debug__:
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1263 # exhaustively verify that the decomposition is correct
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1264 mask = ~((~0) << shift) # i.e., low-bit mask of shift bits
41642 a6eb6acfe04a Merged revisions 55007-55179 via svnmerge from Guido van Rossum <guido@python.org> parents: 36940 diff changeset	1265 for i in range(len(t)):
14834 1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1266 assert t[i] == t2[(t1[i >> shift] << shift) + (i & mask)]
1d627242e29a Fiddled w/ /F's cool new splitbins function: documented it, generalized it Tim Peters <tim.peters@gmail.com> parents: 14825 diff changeset	1267 return best
14825 9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1268
9fd030af2fe0 unicode database compression, step 1: Fredrik Lundh <fredrik@pythonware.com> parents: diff changeset	1269 if __name__ == "__main__":
15376 493ca9e4260e Added 38,642 missing characters to the Unicode database (first-last Fredrik Lundh <fredrik@pythonware.com> parents: 15339 diff changeset	1270 maketables(1)

Mercurial > cpython-lru_cache-weakref

annotate Tools/unicode/makeunicodedata.py @ 73798:6cde416ef03b