Mercurial > cpython-lru_cache-weakref

annotate Tools/unicode/mkstringprep.py @ 87690:66c1c9f32567 default tip

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Add use_weakref option to functools.lru_cache The use_weakref option makes lru_cache create weak references to the function arguments when creating the cache key, which prevents them from staying in the memory forever even after all other references to them are gone. This is a proof of concept for http://bugs.python.org/issue19859
author Radomir Dopieralski <devel@sheep.art.pl>
date Tue, 03 Dec 2013 16:02:48 +0100
parents 8f95d77443da
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
84066
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
1 import re, sys
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
2 from unicodedata import ucd_3_2_0 as unicodedata
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
3
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
4 if sys.maxunicode == 65535:
42715
e4ade16a694f Convert raise statements in Tools/.
Collin Winter <collinw@gmail.com>
parents: 42427
diff changeset
5 raise RuntimeError("need UCS-4 Python")
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
6
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
7 def gen_category(cats):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
8 for i in range(0, 0x110000):
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
9 if unicodedata.category(chr(i)) in cats:
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
10 yield(i)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
11
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
12 def gen_bidirectional(cats):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
13 for i in range(0, 0x110000):
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
14 if unicodedata.bidirectional(chr(i)) in cats:
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
15 yield(i)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
16
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
17 def compact_set(l):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
18 single = []
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
19 tuple = []
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
20 prev = None
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
21 span = 0
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
22 for e in l:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
23 if prev is None:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
24 prev = e
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
25 span = 0
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
26 continue
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
27 if prev+span+1 != e:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
28 if span > 2:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
29 tuple.append((prev,prev+span+1))
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
30 else:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
31 for i in range(prev, prev+span+1):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
32 single.append(i)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
33 prev = e
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
34 span = 0
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
35 else:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
36 span += 1
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
37 if span:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
38 tuple.append((prev,prev+span+1))
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
39 else:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
40 single.append(prev)
84066
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
41 if not single and len(tuple) == 1:
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
42 tuple = "range(%d,%d)" % tuple[0]
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
43 else:
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
44 tuple = " + ".join("list(range(%d,%d))" % t for t in tuple)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
45 if not single:
32125
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
46 return "set(%s)" % tuple
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
47 if not tuple:
84066
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
48 return "set(%r)" % (single,)
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
49 return "set(%r + %s)" % (single, tuple)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
50
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
51 ############## Read the tables in the RFC #######################
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
52
84066
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
53 with open("rfc3454.txt") as f:
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
54 data = f.readlines()
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
55
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
56 tables = []
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
57 curname = None
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
58 for l in data:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
59 l = l.strip()
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
60 if not l:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
61 continue
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
62 # Skip RFC page breaks
84066
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
63 if l.startswith(("Hoffman & Blanchet", "RFC 3454")):
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
64 continue
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
65 # Find start/end lines
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
66 m = re.match("----- (Start|End) Table ([A-Z](.[0-9])+) -----", l)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
67 if m:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
68 if m.group(1) == "Start":
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
69 if curname:
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
70 raise RuntimeError("Double Start", (curname, l))
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
71 curname = m.group(2)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
72 table = {}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
73 tables.append((curname, table))
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
74 continue
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
75 else:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
76 if not curname:
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
77 raise RuntimeError("End without start", l)
84066
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
78 if curname != m.group(2):
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
79 raise RuntimeError("Unexpected end", l)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
80 curname = None
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
81 continue
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
82 if not curname:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
83 continue
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
84 # Now we are in a table
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
85 fields = l.split(";")
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
86 if len(fields) > 1:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
87 # Drop comment field
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
88 fields = fields[:-1]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
89 if len(fields) == 1:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
90 fields = fields[0].split("-")
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
91 if len(fields) > 1:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
92 # range
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
93 try:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
94 start, end = fields
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
95 except ValueError:
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
96 raise RuntimeError("Unpacking problem", l)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
97 else:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
98 start = end = fields[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
99 start = int(start, 16)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
100 end = int(end, 16)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
101 for i in range(start, end+1):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
102 table[i] = i
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
103 else:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
104 code, value = fields
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
105 value = value.strip()
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
106 if value:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
107 value = [int(v, 16) for v in value.split(" ")]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
108 else:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
109 # table B.1
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
110 value = None
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
111 table[int(code, 16)] = value
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
112
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
113 ########### Generate compact Python versions of the tables #############
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
114
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
115 print("""# This file is generated by mkstringprep.py. DO NOT EDIT.
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
116 \"\"\"Library that exposes various tables found in the StringPrep RFC 3454.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
117
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
118 There are two kinds of tables: sets, for which a member test is provided,
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
119 and mappings, for which a mapping function is provided.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
120 \"\"\"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
121
84066
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
122 from unicodedata import ucd_3_2_0 as unicodedata
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
123 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
124
84066
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
125 print("assert unicodedata.unidata_version == %r" % (unicodedata.unidata_version,))
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
126
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
127 # A.1 is the table of unassigned characters
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
128 # XXX Plane 15 PUA is listed as unassigned in Python.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
129 name, table = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
130 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
131 assert name == "A.1"
32125
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
132 table = set(table.keys())
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
133 Cn = set(gen_category(["Cn"]))
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
134
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
135 # FDD0..FDEF are process internal codes
32125
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
136 Cn -= set(range(0xFDD0, 0xFDF0))
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
137 # not a character
32125
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
138 Cn -= set(range(0xFFFE, 0x110000, 0x10000))
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
139 Cn -= set(range(0xFFFF, 0x110000, 0x10000))
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
140
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
141 # assert table == Cn
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
142
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
143 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
144 def in_table_a1(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
145 if unicodedata.category(code) != 'Cn': return False
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
146 c = ord(code)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
147 if 0xFDD0 <= c < 0xFDF0: return False
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
148 return (c & 0xFFFF) not in (0xFFFE, 0xFFFF)
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
149 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
150
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
151 # B.1 cannot easily be derived
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
152 name, table = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
153 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
154 assert name == "B.1"
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
155 table = sorted(table.keys())
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
156 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
157 b1_set = """ + compact_set(table) + """
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
158 def in_table_b1(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
159 return ord(code) in b1_set
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
160 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
161
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
162 # B.2 and B.3 is case folding.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
163 # It takes CaseFolding.txt into account, which is
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
164 # not available in the Python database. Since
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
165 # B.2 is derived from B.3, we process B.3 first.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
166 # B.3 supposedly *is* CaseFolding-3.2.0.txt.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
167
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
168 name, table_b2 = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
169 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
170 assert name == "B.2"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
171
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
172 name, table_b3 = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
173 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
174 assert name == "B.3"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
175
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
176 # B.3 is mostly Python's .lower, except for a number
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
177 # of special cases, e.g. considering canonical forms.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
178
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
179 b3_exceptions = {}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
180
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
181 for k,v in table_b2.items():
84066
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
182 if list(map(ord, chr(k).lower())) != v:
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
183 b3_exceptions[k] = "".join(map(chr,v))
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
184
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
185 b3 = sorted(b3_exceptions.items())
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
186
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
187 print("""
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
188 b3_exceptions = {""")
84066
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
189 for i, kv in enumerate(b3):
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
190 print("0x%x:%a," % kv, end=' ')
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
191 if i % 4 == 3:
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
192 print()
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
193 print("}")
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
194
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
195 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
196 def map_table_b3(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
197 r = b3_exceptions.get(ord(code))
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
198 if r is not None: return r
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
199 return code.lower()
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
200 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
201
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
202 def map_table_b3(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
203 r = b3_exceptions.get(ord(code))
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
204 if r is not None: return r
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
205 return code.lower()
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
206
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
207 # B.2 is case folding for NFKC. This is the same as B.3,
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
208 # except where NormalizeWithKC(Fold(a)) !=
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
209 # NormalizeWithKC(Fold(NormalizeWithKC(Fold(a))))
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
210
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
211 def map_table_b2(a):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
212 al = map_table_b3(a)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
213 b = unicodedata.normalize("NFKC", al)
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
214 bl = "".join([map_table_b3(ch) for ch in b])
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
215 c = unicodedata.normalize("NFKC", bl)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
216 if b != c:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
217 return c
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
218 else:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
219 return al
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
220
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
221 specials = {}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
222 for k,v in table_b2.items():
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
223 if list(map(ord, map_table_b2(chr(k)))) != v:
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
224 specials[k] = v
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
225
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
226 # B.3 should not add any additional special cases
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
227 assert specials == {}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
228
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
229 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
230 def map_table_b2(a):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
231 al = map_table_b3(a)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
232 b = unicodedata.normalize("NFKC", al)
84066
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
233 bl = "".join([map_table_b3(ch) for ch in b])
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
234 c = unicodedata.normalize("NFKC", bl)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
235 if b != c:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
236 return c
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
237 else:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
238 return al
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
239 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
240
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
241 # C.1.1 is a table with a single character
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
242 name, table = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
243 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
244 assert name == "C.1.1"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
245 assert table == {0x20:0x20}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
246
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
247 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
248 def in_table_c11(code):
84066
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
249 return code == " "
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
250 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
251
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
252 # C.1.2 is the rest of all space characters
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
253 name, table = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
254 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
255 assert name == "C.1.2"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
256
32125
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
257 # table = set(table.keys())
84066
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
258 # Zs = set(gen_category(["Zs"])) - {0x20}
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
259 # assert Zs == table
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
260
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
261 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
262 def in_table_c12(code):
84066
8f95d77443da Issue #15239: Make mkstringprep.py work again on Python 3.
Serhiy Storchaka <storchaka@gmail.com>
parents: 47103
diff changeset
263 return unicodedata.category(code) == "Zs" and code != " "
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
264
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
265 def in_table_c11_c12(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
266 return unicodedata.category(code) == "Zs"
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
267 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
268
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
269 # C.2.1 ASCII control characters
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
270 name, table_c21 = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
271 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
272 assert name == "C.2.1"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
273
32125
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
274 Cc = set(gen_category(["Cc"]))
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
275 Cc_ascii = Cc & set(range(128))
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
276 table_c21 = set(table_c21.keys())
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
277 assert Cc_ascii == table_c21
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
278
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
279 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
280 def in_table_c21(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
281 return ord(code) < 128 and unicodedata.category(code) == "Cc"
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
282 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
283
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
284 # C.2.2 Non-ASCII control characters. It also includes
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
285 # a number of characters in category Cf.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
286 name, table_c22 = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
287 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
288 assert name == "C.2.2"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
289
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
290 Cc_nonascii = Cc - Cc_ascii
32125
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
291 table_c22 = set(table_c22.keys())
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
292 assert len(Cc_nonascii - table_c22) == 0
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
293
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
294 specials = list(table_c22 - Cc_nonascii)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
295 specials.sort()
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
296
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
297 print("""c22_specials = """ + compact_set(specials) + """
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
298 def in_table_c22(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
299 c = ord(code)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
300 if c < 128: return False
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
301 if unicodedata.category(code) == "Cc": return True
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
302 return c in c22_specials
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
303
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
304 def in_table_c21_c22(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
305 return unicodedata.category(code) == "Cc" or \\
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
306 ord(code) in c22_specials
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
307 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
308
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
309 # C.3 Private use
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
310 name, table = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
311 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
312 assert name == "C.3"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
313
32125
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
314 Co = set(gen_category(["Co"]))
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
315 assert set(table.keys()) == Co
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
316
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
317 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
318 def in_table_c3(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
319 return unicodedata.category(code) == "Co"
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
320 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
321
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
322 # C.4 Non-character code points, xFFFE, xFFFF
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
323 # plus process internal codes
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
324 name, table = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
325 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
326 assert name == "C.4"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
327
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
328 nonchar = set(range(0xFDD0,0xFDF0))
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
329 nonchar.update(range(0xFFFE,0x110000,0x10000))
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
330 nonchar.update(range(0xFFFF,0x110000,0x10000))
32125
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
331 table = set(table.keys())
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
332 assert table == nonchar
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
333
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
334 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
335 def in_table_c4(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
336 c = ord(code)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
337 if c < 0xFDD0: return False
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
338 if c < 0xFDF0: return True
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
339 return (ord(code) & 0xFFFF) in (0xFFFE, 0xFFFF)
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
340 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
341
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
342 # C.5 Surrogate codes
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
343 name, table = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
344 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
345 assert name == "C.5"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
346
32125
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
347 Cs = set(gen_category(["Cs"]))
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
348 assert set(table.keys()) == Cs
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
349
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
350 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
351 def in_table_c5(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
352 return unicodedata.category(code) == "Cs"
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
353 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
354
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
355 # C.6 Inappropriate for plain text
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
356 name, table = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
357 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
358 assert name == "C.6"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
359
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
360 table = sorted(table.keys())
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
361
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
362 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
363 c6_set = """ + compact_set(table) + """
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
364 def in_table_c6(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
365 return ord(code) in c6_set
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
366 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
367
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
368 # C.7 Inappropriate for canonical representation
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
369 name, table = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
370 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
371 assert name == "C.7"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
372
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
373 table = sorted(table.keys())
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
374
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
375 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
376 c7_set = """ + compact_set(table) + """
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
377 def in_table_c7(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
378 return ord(code) in c7_set
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
379 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
380
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
381 # C.8 Change display properties or are deprecated
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
382 name, table = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
383 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
384 assert name == "C.8"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
385
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
386 table = sorted(table.keys())
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
387
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
388 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
389 c8_set = """ + compact_set(table) + """
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
390 def in_table_c8(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
391 return ord(code) in c8_set
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
392 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
393
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
394 # C.9 Tagging characters
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
395 name, table = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
396 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
397 assert name == "C.9"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
398
47103
63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893.
Georg Brandl <georg@python.org>
parents: 42715
diff changeset
399 table = sorted(table.keys())
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
400
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
401 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
402 c9_set = """ + compact_set(table) + """
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
403 def in_table_c9(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
404 return ord(code) in c9_set
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
405 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
406
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
407 # D.1 Characters with bidirectional property "R" or "AL"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
408 name, table = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
409 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
410 assert name == "D.1"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
411
32125
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
412 RandAL = set(gen_bidirectional(["R","AL"]))
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
413 assert set(table.keys()) == RandAL
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
414
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
415 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
416 def in_table_d1(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
417 return unicodedata.bidirectional(code) in ("R","AL")
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
418 """)
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
419
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
420 # D.2 Characters with bidirectional property "L"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
421 name, table = tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
422 del tables[0]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
423 assert name == "D.2"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
424
32125
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
425 L = set(gen_bidirectional(["L"]))
b9edd3418a31 Applying SF patch #949329 on behalf of Raymond Hettinger.
Armin Rigo <arigo@tunes.org>
parents: 28671
diff changeset
426 assert set(table.keys()) == L
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
427
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
428 print("""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
429 def in_table_d2(code):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
430 return unicodedata.bidirectional(code) == "L"
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 41642
diff changeset
431 """)