Mercurial > cpython-lru_cache-weakref

annotate Tools/unicode/mkstringprep.py @ 87488:124e51c19e4f

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Issue #19734: Also run pip version check in isolated mode
author Nick Coghlan <ncoghlan@gmail.com>
date Sun, 24 Nov 2013 16:49:20 +1000
parents 8f95d77443da
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
84066
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
1 import re, sys
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
2 from unicodedata import ucd_3_2_0 as unicodedata
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
3
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
4 if sys.maxunicode == 65535:
42715
e4ade16a694f Convert raise statements in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42427
diff changeset
5 raise RuntimeError("need UCS-4 Python")
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
6
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
7 def gen_category(cats):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
8 for i in range(0, 0x110000):
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
9 if unicodedata.category(chr(i)) in cats:
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
10 yield(i)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
11
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
12 def gen_bidirectional(cats):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
13 for i in range(0, 0x110000):
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
14 if unicodedata.bidirectional(chr(i)) in cats:
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
15 yield(i)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
16
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
17 def compact_set(l):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
18 single = []
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
19 tuple = []
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
20 prev = None
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
21 span = 0
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
22 for e in l:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
23 if prev is None:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
24 prev = e
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
25 span = 0
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
26 continue
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
27 if prev+span+1 != e:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
28 if span > 2:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
29 tuple.append((prev,prev+span+1))
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
30 else:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
31 for i in range(prev, prev+span+1):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
32 single.append(i)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
33 prev = e
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
34 span = 0
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
35 else:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
36 span += 1
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
37 if span:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
38 tuple.append((prev,prev+span+1))
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
39 else:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
40 single.append(prev)
84066
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
41 if not single and len(tuple) == 1:
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
42 tuple = "range(%d,%d)" % tuple[0]
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
43 else:
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
44 tuple = " + ".join("list(range(%d,%d))" % t for t in tuple)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
45 if not single:
32125
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
46 return "set(%s)" % tuple
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
47 if not tuple:
84066
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
48 return "set(%r)" % (single,)
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
49 return "set(%r + %s)" % (single, tuple)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
50
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
51 ############## Read the tables in the RFC #######################
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
52
84066
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
53 with open("rfc3454.txt") as f:
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
54 data = f.readlines()
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
55
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
56 tables = []
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
57 curname = None
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
58 for l in data:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
59 l = l.strip()
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
60 if not l:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
61 continue
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
62 # Skip RFC page breaks
84066
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
63 if l.startswith(("Hoffman & Blanchet", "RFC 3454")):
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
64 continue
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
65 # Find start/end lines
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
66 m = re.match("----- (Start|End) Table ([A-Z](.[0-9])+) -----", l)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
67 if m:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
68 if m.group(1) == "Start":
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
69 if curname:
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
70 raise RuntimeError("Double Start", (curname, l))
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
71 curname = m.group(2)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
72 table = {}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
73 tables.append((curname, table))
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
74 continue
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
75 else:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
76 if not curname:
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
77 raise RuntimeError("End without start", l)
84066
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
78 if curname != m.group(2):
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
79 raise RuntimeError("Unexpected end", l)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
80 curname = None
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
81 continue
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
82 if not curname:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
83 continue
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
84 # Now we are in a table
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
85 fields = l.split(";")
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
86 if len(fields) > 1:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
87 # Drop comment field
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
88 fields = fields[:-1]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
89 if len(fields) == 1:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
90 fields = fields[0].split("-")
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
91 if len(fields) > 1:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
92 # range
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
93 try:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
94 start, end = fields
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
95 except ValueError:
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
96 raise RuntimeError("Unpacking problem", l)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
97 else:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
98 start = end = fields[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
99 start = int(start, 16)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
100 end = int(end, 16)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
101 for i in range(start, end+1):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
102 table[i] = i
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
103 else:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
104 code, value = fields
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
105 value = value.strip()
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
106 if value:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
107 value = [int(v, 16) for v in value.split(" ")]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
108 else:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
109 # table B.1
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
110 value = None
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
111 table[int(code, 16)] = value
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
112
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
113 ########### Generate compact Python versions of the tables #############
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
114
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
115 print("""# This file is generated by mkstringprep.py. DO NOT EDIT.
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
116 \"\"\"Library that exposes various tables found in the StringPrep RFC 3454.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
117
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
118 There are two kinds of tables: sets, for which a member test is provided,
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
119 and mappings, for which a mapping function is provided.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
120 \"\"\"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
121
84066
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
122 from unicodedata import ucd_3_2_0 as unicodedata
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
123 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
124
84066
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
125 print("assert unicodedata.unidata_version == %r" % (unicodedata.unidata_version,))
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
126
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
127 # A.1 is the table of unassigned characters
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
128 # XXX Plane 15 PUA is listed as unassigned in Python.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
129 name, table = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
130 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
131 assert name == "A.1"
32125
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
132 table = set(table.keys())
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
133 Cn = set(gen_category(["Cn"]))
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
134
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
135 # FDD0..FDEF are process internal codes
32125
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
136 Cn -= set(range(0xFDD0, 0xFDF0))
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
137 # not a character
32125
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
138 Cn -= set(range(0xFFFE, 0x110000, 0x10000))
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
139 Cn -= set(range(0xFFFF, 0x110000, 0x10000))
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
140
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
141 # assert table == Cn
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
142
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
143 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
144 def in_table_a1(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
145 if unicodedata.category(code) != 'Cn': return False
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
146 c = ord(code)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
147 if 0xFDD0 <= c < 0xFDF0: return False
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
148 return (c & 0xFFFF) not in (0xFFFE, 0xFFFF)
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
149 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
150
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
151 # B.1 cannot easily be derived
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
152 name, table = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
153 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
154 assert name == "B.1"
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
155 table = sorted(table.keys())
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
156 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
157 b1_set = """ + compact_set(table) + """
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
158 def in_table_b1(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
159 return ord(code) in b1_set
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
160 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
161
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
162 # B.2 and B.3 is case folding.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
163 # It takes CaseFolding.txt into account, which is
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
164 # not available in the Python database. Since
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
165 # B.2 is derived from B.3, we process B.3 first.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
166 # B.3 supposedly *is* CaseFolding-3.2.0.txt.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
167
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
168 name, table_b2 = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
169 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
170 assert name == "B.2"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
171
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
172 name, table_b3 = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
173 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
174 assert name == "B.3"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
175
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
176 # B.3 is mostly Python's .lower, except for a number
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
177 # of special cases, e.g. considering canonical forms.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
178
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
179 b3_exceptions = {}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
180
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
181 for k,v in table_b2.items():
84066
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
182 if list(map(ord, chr(k).lower())) != v:
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
183 b3_exceptions[k] = "".join(map(chr,v))
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
184
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
185 b3 = sorted(b3_exceptions.items())
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
186
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
187 print("""
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
188 b3_exceptions = {""")
84066
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
189 for i, kv in enumerate(b3):
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
190 print("0x%x:%a," % kv, end=' ')
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
191 if i % 4 == 3:
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
192 print()
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
193 print("}")
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
194
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
195 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
196 def map_table_b3(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
197 r = b3_exceptions.get(ord(code))
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
198 if r is not None: return r
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
199 return code.lower()
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
200 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
201
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
202 def map_table_b3(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
203 r = b3_exceptions.get(ord(code))
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
204 if r is not None: return r
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
205 return code.lower()
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
206
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
207 # B.2 is case folding for NFKC. This is the same as B.3,
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
208 # except where NormalizeWithKC(Fold(a)) !=
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
209 # NormalizeWithKC(Fold(NormalizeWithKC(Fold(a))))
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
210
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
211 def map_table_b2(a):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
212 al = map_table_b3(a)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
213 b = unicodedata.normalize("NFKC", al)
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
214 bl = "".join([map_table_b3(ch) for ch in b])
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
215 c = unicodedata.normalize("NFKC", bl)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
216 if b != c:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
217 return c
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
218 else:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
219 return al
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
220
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
221 specials = {}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
222 for k,v in table_b2.items():
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
223 if list(map(ord, map_table_b2(chr(k)))) != v:
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
224 specials[k] = v
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
225
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
226 # B.3 should not add any additional special cases
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
227 assert specials == {}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
228
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
229 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
230 def map_table_b2(a):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
231 al = map_table_b3(a)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
232 b = unicodedata.normalize("NFKC", al)
84066
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
233 bl = "".join([map_table_b3(ch) for ch in b])
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
234 c = unicodedata.normalize("NFKC", bl)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
235 if b != c:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
236 return c
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
237 else:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
238 return al
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
239 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
240
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
241 # C.1.1 is a table with a single character
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
242 name, table = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
243 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
244 assert name == "C.1.1"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
245 assert table == {0x20:0x20}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
246
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
247 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
248 def in_table_c11(code):
84066
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
249 return code == " "
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
250 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
251
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
252 # C.1.2 is the rest of all space characters
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
253 name, table = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
254 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
255 assert name == "C.1.2"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
256
32125
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
257 # table = set(table.keys())
84066
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
258 # Zs = set(gen_category(["Zs"])) - {0x20}
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
259 # assert Zs == table
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
260
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
261 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
262 def in_table_c12(code):
84066
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
263 return unicodedata.category(code) == "Zs" and code != " "
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
264
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
265 def in_table_c11_c12(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
266 return unicodedata.category(code) == "Zs"
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
267 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
268
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
269 # C.2.1 ASCII control characters
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
270 name, table_c21 = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
271 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
272 assert name == "C.2.1"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
273
32125
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
274 Cc = set(gen_category(["Cc"]))
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
275 Cc_ascii = Cc & set(range(128))
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
276 table_c21 = set(table_c21.keys())
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
277 assert Cc_ascii == table_c21
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
278
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
279 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
280 def in_table_c21(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
281 return ord(code) < 128 and unicodedata.category(code) == "Cc"
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
282 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
283
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
284 # C.2.2 Non-ASCII control characters. It also includes
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
285 # a number of characters in category Cf.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
286 name, table_c22 = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
287 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
288 assert name == "C.2.2"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
289
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
290 Cc_nonascii = Cc - Cc_ascii
32125
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
291 table_c22 = set(table_c22.keys())
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
292 assert len(Cc_nonascii - table_c22) == 0
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
293
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
294 specials = list(table_c22 - Cc_nonascii)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
295 specials.sort()
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
296
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
297 print("""c22_specials = """ + compact_set(specials) + """
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
298 def in_table_c22(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
299 c = ord(code)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
300 if c < 128: return False
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
301 if unicodedata.category(code) == "Cc": return True
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
302 return c in c22_specials
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
303
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
304 def in_table_c21_c22(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
305 return unicodedata.category(code) == "Cc" or \\
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
306 ord(code) in c22_specials
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
307 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
308
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
309 # C.3 Private use
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
310 name, table = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
311 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
312 assert name == "C.3"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
313
32125
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
314 Co = set(gen_category(["Co"]))
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
315 assert set(table.keys()) == Co
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
316
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
317 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
318 def in_table_c3(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
319 return unicodedata.category(code) == "Co"
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
320 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
321
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
322 # C.4 Non-character code points, xFFFE, xFFFF
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
323 # plus process internal codes
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
324 name, table = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
325 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
326 assert name == "C.4"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
327
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
328 nonchar = set(range(0xFDD0,0xFDF0))
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
329 nonchar.update(range(0xFFFE,0x110000,0x10000))
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
330 nonchar.update(range(0xFFFF,0x110000,0x10000))
32125
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
331 table = set(table.keys())
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
332 assert table == nonchar
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
333
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
334 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
335 def in_table_c4(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
336 c = ord(code)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
337 if c < 0xFDD0: return False
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
338 if c < 0xFDF0: return True
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
339 return (ord(code) & 0xFFFF) in (0xFFFE, 0xFFFF)
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
340 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
341
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
342 # C.5 Surrogate codes
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
343 name, table = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
344 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
345 assert name == "C.5"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
346
32125
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
347 Cs = set(gen_category(["Cs"]))
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
348 assert set(table.keys()) == Cs
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
349
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
350 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
351 def in_table_c5(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
352 return unicodedata.category(code) == "Cs"
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
353 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
354
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
355 # C.6 Inappropriate for plain text
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
356 name, table = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
357 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
358 assert name == "C.6"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
359
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
360 table = sorted(table.keys())
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
361
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
362 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
363 c6_set = """ + compact_set(table) + """
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
364 def in_table_c6(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
365 return ord(code) in c6_set
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
366 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
367
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
368 # C.7 Inappropriate for canonical representation
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
369 name, table = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
370 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
371 assert name == "C.7"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
372
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
373 table = sorted(table.keys())
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
374
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
375 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
376 c7_set = """ + compact_set(table) + """
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
377 def in_table_c7(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
378 return ord(code) in c7_set
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
379 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
380
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
381 # C.8 Change display properties or are deprecated
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
382 name, table = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
383 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
384 assert name == "C.8"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
385
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
386 table = sorted(table.keys())
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
387
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
388 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
389 c8_set = """ + compact_set(table) + """
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
390 def in_table_c8(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
391 return ord(code) in c8_set
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
392 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
393
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
394 # C.9 Tagging characters
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
395 name, table = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
396 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
397 assert name == "C.9"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
398
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
399 table = sorted(table.keys())
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
400
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
401 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
402 c9_set = """ + compact_set(table) + """
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
403 def in_table_c9(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
404 return ord(code) in c9_set
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
405 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
406
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
407 # D.1 Characters with bidirectional property "R" or "AL"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
408 name, table = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
409 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
410 assert name == "D.1"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
411
32125
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
412 RandAL = set(gen_bidirectional(["R","AL"]))
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
413 assert set(table.keys()) == RandAL
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
414
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
415 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
416 def in_table_d1(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
417 return unicodedata.bidirectional(code) in ("R","AL")
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
418 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
419
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
420 # D.2 Characters with bidirectional property "L"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
421 name, table = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
422 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
423 assert name == "D.2"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
424
32125
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
425 L = set(gen_bidirectional(["L"]))
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
426 assert set(table.keys()) == L
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
427
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
428 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
429 def in_table_d2(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
430 return unicodedata.bidirectional(code) == "L"
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
431 """)