Mercurial > cpython-lru_cache-weakref

annotate Lib/encodings/punycode.py @ 79675:c0a2294de787 2.7

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Issue #16199: Backport 2to3 typo fix from 3.2.
author Chris Jerdonek <chris.jerdonek@gmail.com>
date Thu, 11 Oct 2012 16:15:46 -0700
parents 992126a29e0f
children f340cb045bf9
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
1 # -*- coding: iso-8859-1 -*-
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
2 """ Codec for the Punicode encoding, as specified in RFC 3492
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
3
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
4 Written by Martin v. Löwis.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
5 """
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
6
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
7 import codecs
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
8
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
9 ##################### Encoding #####################################
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
10
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
11 def segregate(str):
28774
c9bee11eaa8b Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 28671
diff changeset
12 """3.1 Basic code point segregation"""
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
13 base = []
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
14 extended = {}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
15 for c in str:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
16 if ord(c) < 128:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
17 base.append(c)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
18 else:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
19 extended[c] = 1
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
20 extended = extended.keys()
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
21 extended.sort()
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
22 return "".join(base).encode("ascii"),extended
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
23
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
24 def selective_len(str, max):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
25 """Return the length of str, considering only characters below max."""
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
26 res = 0
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
27 for c in str:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
28 if ord(c) < max:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
29 res += 1
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
30 return res
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
31
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
32 def selective_find(str, char, index, pos):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
33 """Return a pair (index, pos), indicating the next occurrence of
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
34 char in str. index is the position of the character considering
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
35 only ordinals up to and including char, and pos is the position in
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
36 the full string. index/pos is the starting position in the full
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
37 string."""
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
38
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
39 l = len(str)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
40 while 1:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
41 pos += 1
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
42 if pos == l:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
43 return (-1, -1)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
44 c = str[pos]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
45 if c == char:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
46 return index+1, pos
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
47 elif c < char:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
48 index += 1
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
49
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
50 def insertion_unsort(str, extended):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
51 """3.2 Insertion unsort coding"""
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
52 oldchar = 0x80
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
53 result = []
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
54 oldindex = -1
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
55 for c in extended:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
56 index = pos = -1
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
57 char = ord(c)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
58 curlen = selective_len(str, char)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
59 delta = (curlen+1) * (char - oldchar)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
60 while 1:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
61 index,pos = selective_find(str,c,index,pos)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
62 if index == -1:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
63 break
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
64 delta += index - oldindex
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
65 result.append(delta-1)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
66 oldindex = index
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
67 delta = 0
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
68 oldchar = char
28774
c9bee11eaa8b Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 28671
diff changeset
69
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
70 return result
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
71
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
72 def T(j, bias):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
73 # Punycode parameters: tmin = 1, tmax = 26, base = 36
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
74 res = 36 * (j + 1) - bias
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
75 if res < 1: return 1
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
76 if res > 26: return 26
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
77 return res
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
78
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
79 digits = "abcdefghijklmnopqrstuvwxyz0123456789"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
80 def generate_generalized_integer(N, bias):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
81 """3.3 Generalized variable-length integers"""
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
82 result = []
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
83 j = 0
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
84 while 1:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
85 t = T(j, bias)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
86 if N < t:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
87 result.append(digits[N])
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
88 return result
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
89 result.append(digits[t + ((N - t) % (36 - t))])
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
90 N = (N - t) // (36 - t)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
91 j += 1
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
92
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
93 def adapt(delta, first, numchars):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
94 if first:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
95 delta //= 700
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
96 else:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
97 delta //= 2
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
98 delta += delta // numchars
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
99 # ((base - tmin) * tmax) // 2 == 455
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
100 divisions = 0
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
101 while delta > 455:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
102 delta = delta // 35 # base - tmin
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
103 divisions += 36
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
104 bias = divisions + (36 * delta // (delta + 38))
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
105 return bias
28774
c9bee11eaa8b Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 28671
diff changeset
106
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
107
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
108 def generate_integers(baselen, deltas):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
109 """3.4 Bias adaptation"""
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
110 # Punycode parameters: initial bias = 72, damp = 700, skew = 38
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
111 result = []
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
112 bias = 72
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
113 for points, delta in enumerate(deltas):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
114 s = generate_generalized_integer(delta, bias)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
115 result.extend(s)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
116 bias = adapt(delta, points==0, baselen+points+1)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
117 return "".join(result)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
118
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
119 def punycode_encode(text):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
120 base, extended = segregate(text)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
121 base = base.encode("ascii")
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
122 deltas = insertion_unsort(text, extended)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
123 extended = generate_integers(len(base), deltas)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
124 if base:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
125 return base + "-" + extended
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
126 return extended
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
127
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
128 ##################### Decoding #####################################
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
129
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
130 def decode_generalized_number(extended, extpos, bias, errors):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
131 """3.3 Generalized variable-length integers"""
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
132 result = 0
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
133 w = 1
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
134 j = 0
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
135 while 1:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
136 try:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
137 char = ord(extended[extpos])
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
138 except IndexError:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
139 if errors == "strict":
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
140 raise UnicodeError, "incomplete punicode string"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
141 return extpos + 1, None
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
142 extpos += 1
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
143 if 0x41 <= char <= 0x5A: # A-Z
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
144 digit = char - 0x41
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
145 elif 0x30 <= char <= 0x39:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
146 digit = char - 22 # 0x30-26
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
147 elif errors == "strict":
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
148 raise UnicodeError("Invalid extended code point '%s'"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
149 % extended[extpos])
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
150 else:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
151 return extpos, None
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
152 t = T(j, bias)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
153 result += digit * w
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
154 if digit < t:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
155 return extpos, result
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
156 w = w * (36 - t)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
157 j += 1
28774
c9bee11eaa8b Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 28671
diff changeset
158
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
159
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
160 def insertion_sort(base, extended, errors):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
161 """3.2 Insertion unsort coding"""
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
162 char = 0x80
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
163 pos = -1
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
164 bias = 72
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
165 extpos = 0
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
166 while extpos < len(extended):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
167 newpos, delta = decode_generalized_number(extended, extpos,
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
168 bias, errors)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
169 if delta is None:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
170 # There was an error in decoding. We can't continue because
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
171 # synchronization is lost.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
172 return base
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
173 pos += delta+1
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
174 char += pos // (len(base) + 1)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
175 if char > 0x10FFFF:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
176 if errors == "strict":
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
177 raise UnicodeError, ("Invalid character U+%x" % char)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
178 char = ord('?')
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
179 pos = pos % (len(base) + 1)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
180 base = base[:pos] + unichr(char) + base[pos:]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
181 bias = adapt(delta, (extpos == 0), len(base))
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
182 extpos = newpos
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
183 return base
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
184
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
185 def punycode_decode(text, errors):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
186 pos = text.rfind("-")
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
187 if pos == -1:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
188 base = ""
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
189 extended = text
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
190 else:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
191 base = text[:pos]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
192 extended = text[pos+1:]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
193 base = unicode(base, "ascii", errors)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
194 extended = extended.upper()
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
195 return insertion_sort(base, extended, errors)
28774
c9bee11eaa8b Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 28671
diff changeset
196
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
197 ### Codec APIs
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
198
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
199 class Codec(codecs.Codec):
36996
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 28774
diff changeset
200
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
201 def encode(self,input,errors='strict'):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
202 res = punycode_encode(input)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
203 return res, len(input)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
204
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
205 def decode(self,input,errors='strict'):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
206 if errors not in ('strict', 'replace', 'ignore'):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
207 raise UnicodeError, "Unsupported error handling "+errors
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
208 res = punycode_decode(input, errors)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
209 return res, len(input)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
210
36996
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 28774
diff changeset
211 class IncrementalEncoder(codecs.IncrementalEncoder):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 28774
diff changeset
212 def encode(self, input, final=False):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 28774
diff changeset
213 return punycode_encode(input)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 28774
diff changeset
214
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 28774
diff changeset
215 class IncrementalDecoder(codecs.IncrementalDecoder):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 28774
diff changeset
216 def decode(self, input, final=False):
38793
992126a29e0f errors is an attribute in the incremental decoder
Walter Dörwald <walter@livinglogic.de>
parents: 36996
diff changeset
217 if self.errors not in ('strict', 'replace', 'ignore'):
992126a29e0f errors is an attribute in the incremental decoder
Walter Dörwald <walter@livinglogic.de>
parents: 36996
diff changeset
218 raise UnicodeError, "Unsupported error handling "+self.errors
992126a29e0f errors is an attribute in the incremental decoder
Walter Dörwald <walter@livinglogic.de>
parents: 36996
diff changeset
219 return punycode_decode(input, self.errors)
36996
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 28774
diff changeset
220
28671
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
221 class StreamWriter(Codec,codecs.StreamWriter):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
222 pass
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
223
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
224 class StreamReader(Codec,codecs.StreamReader):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
225 pass
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
226
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
227 ### encodings module API
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
228
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications).
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
229 def getregentry():
36996
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 28774
diff changeset
230 return codecs.CodecInfo(
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 28774
diff changeset
231 name='punycode',
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 28774
diff changeset
232 encode=Codec().encode,
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 28774
diff changeset
233 decode=Codec().decode,
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 28774
diff changeset
234 incrementalencoder=IncrementalEncoder,
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 28774
diff changeset
235 incrementaldecoder=IncrementalDecoder,
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 28774
diff changeset
236 streamwriter=StreamWriter,
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 28774
diff changeset
237 streamreader=StreamReader,
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 28774
diff changeset
238 )