cpython-lru_cache-weakref: Lib/test/test

annotate Lib/test/test_codecs.py @ 58765:2ba7677bfa95 legacy-trunk

testCopysign was defined twice in test_math; combine the definitions

author	Mark Dickinson <dickinsm@gmail.com>
date	Sat, 06 Feb 2010 23:11:25 +0000
parents	6ffcdafa0118
children	7e3c47354da3

rev	line source
24364 aa0d2d51fbde Get rid of relative imports in all unittests. Now anything that Barry Warsaw <barry@python.org> parents: 20013 diff changeset	1 from test import test_support
aa0d2d51fbde Get rid of relative imports in all unittests. Now anything that Barry Warsaw <barry@python.org> parents: 20013 diff changeset	2 import unittest
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	3 import codecs
37077 38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	4 import sys, StringIO, _testcapi
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	5
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	6 class Queue(object):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	7 """
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	8 queue: write bytes at one end, read bytes from the other end
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	9 """
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	10 def __init__(self):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	11 self._buffer = ""
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	12
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	13 def write(self, chars):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	14 self._buffer += chars
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	15
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	16 def read(self, size=-1):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	17 if size<0:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	18 s = self._buffer
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	19 self._buffer = ""
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	20 return s
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	21 else:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	22 s = self._buffer[:size]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	23 self._buffer = self._buffer[size:]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	24 return s
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	25
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	26 class ReadTest(unittest.TestCase):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	27 def check_partial(self, input, partialresults):
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	28 # get a StreamReader for the encoding and feed the bytestring version
43903 6aa8e135d9f8 Fix typo in comment. Walter Dörwald <walter@livinglogic.de> parents: 42649 diff changeset	29 # of input to the reader byte by byte. Read everything available from
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	30 # the StreamReader and check that the results equal the appropriate
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	31 # entries from partialresults.
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	32 q = Queue()
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	33 r = codecs.getreader(self.encoding)(q)
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	34 result = u""
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	35 for (c, partialresult) in zip(input.encode(self.encoding), partialresults):
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	36 q.write(c)
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	37 result += r.read()
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	38 self.assertEqual(result, partialresult)
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	39 # check that there's nothing left in the buffers
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	40 self.assertEqual(r.read(), u"")
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	41 self.assertEqual(r.bytebuffer, "")
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	42 self.assertEqual(r.charbuffer, u"")
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	43
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	44 # do the check again, this time using a incremental decoder
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	45 d = codecs.getincrementaldecoder(self.encoding)()
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	46 result = u""
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	47 for (c, partialresult) in zip(input.encode(self.encoding), partialresults):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	48 result += d.decode(c)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	49 self.assertEqual(result, partialresult)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	50 # check that there's nothing left in the buffers
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	51 self.assertEqual(d.decode("", True), u"")
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	52 self.assertEqual(d.buffer, "")
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	53
43932 e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	54 # Check whether the reset method works properly
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	55 d.reset()
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	56 result = u""
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	57 for (c, partialresult) in zip(input.encode(self.encoding), partialresults):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	58 result += d.decode(c)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	59 self.assertEqual(result, partialresult)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	60 # check that there's nothing left in the buffers
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	61 self.assertEqual(d.decode("", True), u"")
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	62 self.assertEqual(d.buffer, "")
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	63
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	64 # check iterdecode()
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	65 encoded = input.encode(self.encoding)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	66 self.assertEqual(
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	67 input,
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	68 u"".join(codecs.iterdecode(encoded, self.encoding))
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	69 )
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	70
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	71 def test_readline(self):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	72 def getreader(input):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	73 stream = StringIO.StringIO(input.encode(self.encoding))
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	74 return codecs.getreader(self.encoding)(stream)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	75
36855 204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	76 def readalllines(input, keepends=True, size=None):
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	77 reader = getreader(input)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	78 lines = []
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	79 while True:
36855 204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	80 line = reader.readline(size=size, keepends=keepends)
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	81 if not line:
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	82 break
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	83 lines.append(line)
36855 204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	84 return "\|".join(lines)
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	85
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	86 s = u"foo\nbar\r\nbaz\rspam\u2028eggs"
36855 204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	87 sexpected = u"foo\n\|bar\r\n\|baz\r\|spam\u2028\|eggs"
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	88 sexpectednoends = u"foo\|bar\|baz\|spam\|eggs"
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	89 self.assertEqual(readalllines(s, True), sexpected)
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	90 self.assertEqual(readalllines(s, False), sexpectednoends)
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	91 self.assertEqual(readalllines(s, True, 10), sexpected)
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	92 self.assertEqual(readalllines(s, False, 10), sexpectednoends)
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	93
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	94 # Test long lines (multiple calls to read() in readline())
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	95 vw = []
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	96 vwo = []
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	97 for (i, lineend) in enumerate(u"\n \r\n \r \u2028".split()):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	98 vw.append((i200)u"\3042" + lineend)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	99 vwo.append((i200)u"\3042")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	100 self.assertEqual(readalllines("".join(vw), True), "".join(vw))
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	101 self.assertEqual(readalllines("".join(vw), False),"".join(vwo))
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	102
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	103 # Test lines where the first read might end with \r, so the
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	104 # reader has to look ahead whether this is a lone \r or a \r\n
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	105 for size in xrange(80):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	106 for lineend in u"\n \r\n \r \u2028".split():
34865 b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	107 s = 10(sizeu"a" + lineend + u"xxx\n")
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	108 reader = getreader(s)
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	109 for i in xrange(10):
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	110 self.assertEqual(
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	111 reader.readline(keepends=True),
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	112 size*u"a" + lineend,
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	113 )
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	114 reader = getreader(s)
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	115 for i in xrange(10):
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	116 self.assertEqual(
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	117 reader.readline(keepends=False),
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	118 size*u"a",
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	119 )
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	120
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	121 def test_bug1175396(self):
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	122 s = [
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	123 '<%!--===================================================\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	124 ' BLOG index page: show recent articles,\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	125 ' today\'s articles, or articles of a specific date.\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	126 '========================================================--%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	127 '<%@inputencoding="ISO-8859-1"%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	128 '<%@pagetemplate=TEMPLATE.y%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	129 '<%@import=import frog.util, frog%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	130 '<%@import=import frog.objects%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	131 '<%@import=from frog.storageerrors import StorageError%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	132 '<%\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	133 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	134 'import logging\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	135 'log=logging.getLogger("Snakelets.logger")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	136 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	137 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	138 'user=self.SessionCtx.user\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	139 'storageEngine=self.SessionCtx.storageEngine\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	140 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	141 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	142 'def readArticlesFromDate(date, count=None):\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	143 ' entryids=storageEngine.listBlogEntries(date)\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	144 ' entryids.reverse() # descending\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	145 ' if count:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	146 ' entryids=entryids[:count]\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	147 ' try:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	148 ' return [ frog.objects.BlogEntry.load(storageEngine, date, Id) for Id in entryids ]\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	149 ' except StorageError,x:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	150 ' log.error("Error loading articles: "+str(x))\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	151 ' self.abort("cannot load articles")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	152 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	153 'showdate=None\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	154 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	155 'arg=self.Request.getArg()\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	156 'if arg=="today":\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	157 ' #-------------------- TODAY\'S ARTICLES\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	158 ' self.write("<h2>Today\'s articles</h2>")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	159 ' showdate = frog.util.isodatestr() \r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	160 ' entries = readArticlesFromDate(showdate)\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	161 'elif arg=="active":\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	162 ' #-------------------- ACTIVE ARTICLES redirect\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	163 ' self.Yredirect("active.y")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	164 'elif arg=="login":\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	165 ' #-------------------- LOGIN PAGE redirect\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	166 ' self.Yredirect("login.y")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	167 'elif arg=="date":\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	168 ' #-------------------- ARTICLES OF A SPECIFIC DATE\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	169 ' showdate = self.Request.getParameter("date")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	170 ' self.write("<h2>Articles written on %s</h2>"% frog.util.mediumdatestr(showdate))\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	171 ' entries = readArticlesFromDate(showdate)\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	172 'else:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	173 ' #-------------------- RECENT ARTICLES\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	174 ' self.write("<h2>Recent articles</h2>")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	175 ' dates=storageEngine.listBlogEntryDates()\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	176 ' if dates:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	177 ' entries=[]\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	178 ' SHOWAMOUNT=10\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	179 ' for showdate in dates:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	180 ' entries.extend( readArticlesFromDate(showdate, SHOWAMOUNT-len(entries)) )\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	181 ' if len(entries)>=SHOWAMOUNT:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	182 ' break\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	183 ' \r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	184 ]
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	185 stream = StringIO.StringIO("".join(s).encode(self.encoding))
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	186 reader = codecs.getreader(self.encoding)(stream)
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	187 for (i, line) in enumerate(reader):
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	188 self.assertEqual(line, s[i])
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	189
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	190 def test_readlinequeue(self):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	191 q = Queue()
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	192 writer = codecs.getwriter(self.encoding)(q)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	193 reader = codecs.getreader(self.encoding)(q)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	194
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	195 # No lineends
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	196 writer.write(u"foo\r")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	197 self.assertEqual(reader.readline(keepends=False), u"foo")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	198 writer.write(u"\nbar\r")
34925 1331c7799640 Update test to the current readline() behaviour. Walter Dörwald <walter@livinglogic.de> parents: 34865 diff changeset	199 self.assertEqual(reader.readline(keepends=False), u"")
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	200 self.assertEqual(reader.readline(keepends=False), u"bar")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	201 writer.write(u"baz")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	202 self.assertEqual(reader.readline(keepends=False), u"baz")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	203 self.assertEqual(reader.readline(keepends=False), u"")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	204
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	205 # Lineends
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	206 writer.write(u"foo\r")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	207 self.assertEqual(reader.readline(keepends=True), u"foo\r")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	208 writer.write(u"\nbar\r")
34925 1331c7799640 Update test to the current readline() behaviour. Walter Dörwald <walter@livinglogic.de> parents: 34865 diff changeset	209 self.assertEqual(reader.readline(keepends=True), u"\n")
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	210 self.assertEqual(reader.readline(keepends=True), u"bar\r")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	211 writer.write(u"baz")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	212 self.assertEqual(reader.readline(keepends=True), u"baz")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	213 self.assertEqual(reader.readline(keepends=True), u"")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	214 writer.write(u"foo\r\n")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	215 self.assertEqual(reader.readline(keepends=True), u"foo\r\n")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	216
34384 a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	217 def test_bug1098990_a(self):
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	218 s1 = u"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	219 s2 = u"offending line: ladfj askldfj klasdj fskla dfzaskdj fasklfj laskd fjasklfzzzzaa%whereisthis!!!\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	220 s3 = u"next line.\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	221
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	222 s = (s1+s2+s3).encode(self.encoding)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	223 stream = StringIO.StringIO(s)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	224 reader = codecs.getreader(self.encoding)(stream)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	225 self.assertEqual(reader.readline(), s1)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	226 self.assertEqual(reader.readline(), s2)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	227 self.assertEqual(reader.readline(), s3)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	228 self.assertEqual(reader.readline(), u"")
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	229
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	230 def test_bug1098990_b(self):
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	231 s1 = u"aaaaaaaaaaaaaaaaaaaaaaaa\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	232 s2 = u"bbbbbbbbbbbbbbbbbbbbbbbb\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	233 s3 = u"stillokay:bbbbxx\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	234 s4 = u"broken!!!!badbad\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	235 s5 = u"againokay.\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	236
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	237 s = (s1+s2+s3+s4+s5).encode(self.encoding)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	238 stream = StringIO.StringIO(s)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	239 reader = codecs.getreader(self.encoding)(stream)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	240 self.assertEqual(reader.readline(), s1)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	241 self.assertEqual(reader.readline(), s2)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	242 self.assertEqual(reader.readline(), s3)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	243 self.assertEqual(reader.readline(), s4)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	244 self.assertEqual(reader.readline(), s5)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	245 self.assertEqual(reader.readline(), u"")
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	246
42649 c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	247 class UTF32Test(ReadTest):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	248 encoding = "utf-32"
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	249
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	250 spamle = ('\xff\xfe\x00\x00'
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	251 's\x00\x00\x00p\x00\x00\x00a\x00\x00\x00m\x00\x00\x00'
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	252 's\x00\x00\x00p\x00\x00\x00a\x00\x00\x00m\x00\x00\x00')
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	253 spambe = ('\x00\x00\xfe\xff'
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	254 '\x00\x00\x00s\x00\x00\x00p\x00\x00\x00a\x00\x00\x00m'
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	255 '\x00\x00\x00s\x00\x00\x00p\x00\x00\x00a\x00\x00\x00m')
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	256
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	257 def test_only_one_bom(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	258 _,_,reader,writer = codecs.lookup(self.encoding)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	259 # encode some stream
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	260 s = StringIO.StringIO()
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	261 f = writer(s)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	262 f.write(u"spam")
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	263 f.write(u"spam")
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	264 d = s.getvalue()
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	265 # check whether there is exactly one BOM in it
55002 6768b2350c57 convert usage of fail* to assert* Benjamin Peterson <benjamin@python.org> parents: 53855 diff changeset	266 self.assertTrue(d == self.spamle or d == self.spambe)
42649 c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	267 # try to read it back
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	268 s = StringIO.StringIO(d)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	269 f = reader(s)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	270 self.assertEquals(f.read(), u"spamspam")
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	271
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	272 def test_badbom(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	273 s = StringIO.StringIO(4*"\xff")
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	274 f = codecs.getreader(self.encoding)(s)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	275 self.assertRaises(UnicodeError, f.read)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	276
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	277 s = StringIO.StringIO(8*"\xff")
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	278 f = codecs.getreader(self.encoding)(s)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	279 self.assertRaises(UnicodeError, f.read)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	280
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	281 def test_partial(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	282 self.check_partial(
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	283 u"\x00\xff\u0100\uffff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	284 [
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	285 u"", # first byte of BOM read
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	286 u"", # second byte of BOM read
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	287 u"", # third byte of BOM read
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	288 u"", # fourth byte of BOM read => byteorder known
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	289 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	290 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	291 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	292 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	293 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	294 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	295 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	296 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	297 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	298 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	299 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	300 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	301 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	302 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	303 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	304 u"\x00\xff\u0100\uffff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	305 ]
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	306 )
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	307
56033 38a52b672429 Issue #6922: Fix an infinite loop when trying to decode an invalid Georg Brandl <georg@python.org> parents: 55002 diff changeset	308 def test_handlers(self):
38a52b672429 Issue #6922: Fix an infinite loop when trying to decode an invalid Georg Brandl <georg@python.org> parents: 55002 diff changeset	309 self.assertEqual((u'\ufffd', 1),
38a52b672429 Issue #6922: Fix an infinite loop when trying to decode an invalid Georg Brandl <georg@python.org> parents: 55002 diff changeset	310 codecs.utf_32_decode('\x01', 'replace', True))
38a52b672429 Issue #6922: Fix an infinite loop when trying to decode an invalid Georg Brandl <georg@python.org> parents: 55002 diff changeset	311 self.assertEqual((u'', 1),
38a52b672429 Issue #6922: Fix an infinite loop when trying to decode an invalid Georg Brandl <georg@python.org> parents: 55002 diff changeset	312 codecs.utf_32_decode('\x01', 'ignore', True))
38a52b672429 Issue #6922: Fix an infinite loop when trying to decode an invalid Georg Brandl <georg@python.org> parents: 55002 diff changeset	313
42649 c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	314 def test_errors(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	315 self.assertRaises(UnicodeDecodeError, codecs.utf_32_decode,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	316 "\xff", "strict", True)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	317
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	318 class UTF32LETest(ReadTest):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	319 encoding = "utf-32-le"
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	320
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	321 def test_partial(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	322 self.check_partial(
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	323 u"\x00\xff\u0100\uffff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	324 [
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	325 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	326 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	327 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	328 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	329 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	330 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	331 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	332 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	333 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	334 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	335 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	336 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	337 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	338 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	339 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	340 u"\x00\xff\u0100\uffff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	341 ]
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	342 )
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	343
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	344 def test_simple(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	345 self.assertEqual(u"\U00010203".encode(self.encoding), "\x03\x02\x01\x00")
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	346
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	347 def test_errors(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	348 self.assertRaises(UnicodeDecodeError, codecs.utf_32_le_decode,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	349 "\xff", "strict", True)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	350
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	351 class UTF32BETest(ReadTest):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	352 encoding = "utf-32-be"
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	353
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	354 def test_partial(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	355 self.check_partial(
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	356 u"\x00\xff\u0100\uffff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	357 [
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	358 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	359 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	360 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	361 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	362 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	363 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	364 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	365 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	366 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	367 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	368 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	369 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	370 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	371 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	372 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	373 u"\x00\xff\u0100\uffff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	374 ]
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	375 )
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	376
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	377 def test_simple(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	378 self.assertEqual(u"\U00010203".encode(self.encoding), "\x00\x01\x02\x03")
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	379
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	380 def test_errors(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	381 self.assertRaises(UnicodeDecodeError, codecs.utf_32_be_decode,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	382 "\xff", "strict", True)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	383
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	384 class UTF16Test(ReadTest):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	385 encoding = "utf-16"
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	386
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	387 spamle = '\xff\xfes\x00p\x00a\x00m\x00s\x00p\x00a\x00m\x00'
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	388 spambe = '\xfe\xff\x00s\x00p\x00a\x00m\x00s\x00p\x00a\x00m'
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	389
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	390 def test_only_one_bom(self):
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	391 _,_,reader,writer = codecs.lookup(self.encoding)
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	392 # encode some stream
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	393 s = StringIO.StringIO()
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	394 f = writer(s)
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	395 f.write(u"spam")
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	396 f.write(u"spam")
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	397 d = s.getvalue()
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	398 # check whether there is exactly one BOM in it
55002 6768b2350c57 convert usage of fail* to assert* Benjamin Peterson <benjamin@python.org> parents: 53855 diff changeset	399 self.assertTrue(d == self.spamle or d == self.spambe)
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	400 # try to read it back
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	401 s = StringIO.StringIO(d)
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	402 f = reader(s)
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	403 self.assertEquals(f.read(), u"spamspam")
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	404
34538 37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	405 def test_badbom(self):
37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	406 s = StringIO.StringIO("\xff\xff")
34562 7a1277fad04e Fix stupid typo: Don't read from a writer. Walter Dörwald <walter@livinglogic.de> parents: 34538 diff changeset	407 f = codecs.getreader(self.encoding)(s)
34538 37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	408 self.assertRaises(UnicodeError, f.read)
37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	409
37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	410 s = StringIO.StringIO("\xff\xff\xff\xff")
34562 7a1277fad04e Fix stupid typo: Don't read from a writer. Walter Dörwald <walter@livinglogic.de> parents: 34538 diff changeset	411 f = codecs.getreader(self.encoding)(s)
34538 37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	412 self.assertRaises(UnicodeError, f.read)
37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	413
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	414 def test_partial(self):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	415 self.check_partial(
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	416 u"\x00\xff\u0100\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	417 [
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	418 u"", # first byte of BOM read
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	419 u"", # second byte of BOM read => byteorder known
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	420 u"",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	421 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	422 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	423 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	424 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	425 u"\x00\xff\u0100",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	426 u"\x00\xff\u0100",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	427 u"\x00\xff\u0100\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	428 ]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	429 )
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	430
56033 38a52b672429 Issue #6922: Fix an infinite loop when trying to decode an invalid Georg Brandl <georg@python.org> parents: 55002 diff changeset	431 def test_handlers(self):
38a52b672429 Issue #6922: Fix an infinite loop when trying to decode an invalid Georg Brandl <georg@python.org> parents: 55002 diff changeset	432 self.assertEqual((u'\ufffd', 1),
38a52b672429 Issue #6922: Fix an infinite loop when trying to decode an invalid Georg Brandl <georg@python.org> parents: 55002 diff changeset	433 codecs.utf_16_decode('\x01', 'replace', True))
38a52b672429 Issue #6922: Fix an infinite loop when trying to decode an invalid Georg Brandl <georg@python.org> parents: 55002 diff changeset	434 self.assertEqual((u'', 1),
38a52b672429 Issue #6922: Fix an infinite loop when trying to decode an invalid Georg Brandl <georg@python.org> parents: 55002 diff changeset	435 codecs.utf_16_decode('\x01', 'ignore', True))
38a52b672429 Issue #6922: Fix an infinite loop when trying to decode an invalid Georg Brandl <georg@python.org> parents: 55002 diff changeset	436
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	437 def test_errors(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	438 self.assertRaises(UnicodeDecodeError, codecs.utf_16_decode, "\xff", "strict", True)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	439
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	440 class UTF16LETest(ReadTest):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	441 encoding = "utf-16-le"
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	442
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	443 def test_partial(self):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	444 self.check_partial(
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	445 u"\x00\xff\u0100\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	446 [
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	447 u"",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	448 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	449 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	450 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	451 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	452 u"\x00\xff\u0100",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	453 u"\x00\xff\u0100",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	454 u"\x00\xff\u0100\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	455 ]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	456 )
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	457
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	458 def test_errors(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	459 self.assertRaises(UnicodeDecodeError, codecs.utf_16_le_decode, "\xff", "strict", True)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	460
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	461 class UTF16BETest(ReadTest):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	462 encoding = "utf-16-be"
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	463
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	464 def test_partial(self):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	465 self.check_partial(
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	466 u"\x00\xff\u0100\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	467 [
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	468 u"",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	469 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	470 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	471 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	472 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	473 u"\x00\xff\u0100",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	474 u"\x00\xff\u0100",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	475 u"\x00\xff\u0100\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	476 ]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	477 )
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	478
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	479 def test_errors(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	480 self.assertRaises(UnicodeDecodeError, codecs.utf_16_be_decode, "\xff", "strict", True)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	481
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	482 class UTF8Test(ReadTest):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	483 encoding = "utf-8"
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	484
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	485 def test_partial(self):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	486 self.check_partial(
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	487 u"\x00\xff\u07ff\u0800\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	488 [
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	489 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	490 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	491 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	492 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	493 u"\x00\xff\u07ff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	494 u"\x00\xff\u07ff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	495 u"\x00\xff\u07ff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	496 u"\x00\xff\u07ff\u0800",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	497 u"\x00\xff\u07ff\u0800",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	498 u"\x00\xff\u07ff\u0800",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	499 u"\x00\xff\u07ff\u0800\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	500 ]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	501 )
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	502
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	503 class UTF7Test(ReadTest):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	504 encoding = "utf-7"
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	505
43932 e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	506 def test_partial(self):
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	507 self.check_partial(
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	508 u"a+-b",
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	509 [
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	510 u"a",
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	511 u"a",
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	512 u"a+",
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	513 u"a+-",
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	514 u"a+-b",
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	515 ]
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	516 )
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	517
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	518 class UTF16ExTest(unittest.TestCase):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	519
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	520 def test_errors(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	521 self.assertRaises(UnicodeDecodeError, codecs.utf_16_ex_decode, "\xff", "strict", 0, True)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	522
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	523 def test_bad_args(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	524 self.assertRaises(TypeError, codecs.utf_16_ex_decode)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	525
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	526 class ReadBufferTest(unittest.TestCase):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	527
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	528 def test_array(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	529 import array
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	530 self.assertEqual(
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	531 codecs.readbuffer_encode(array.array("c", "spam")),
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	532 ("spam", 4)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	533 )
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	534
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	535 def test_empty(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	536 self.assertEqual(codecs.readbuffer_encode(""), ("", 0))
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	537
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	538 def test_bad_args(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	539 self.assertRaises(TypeError, codecs.readbuffer_encode)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	540 self.assertRaises(TypeError, codecs.readbuffer_encode, 42)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	541
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	542 class CharBufferTest(unittest.TestCase):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	543
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	544 def test_string(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	545 self.assertEqual(codecs.charbuffer_encode("spam"), ("spam", 4))
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	546
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	547 def test_empty(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	548 self.assertEqual(codecs.charbuffer_encode(""), ("", 0))
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	549
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	550 def test_bad_args(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	551 self.assertRaises(TypeError, codecs.charbuffer_encode)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	552 self.assertRaises(TypeError, codecs.charbuffer_encode, 42)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	553
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	554 class UTF8SigTest(ReadTest):
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	555 encoding = "utf-8-sig"
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	556
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	557 def test_partial(self):
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	558 self.check_partial(
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	559 u"\ufeff\x00\xff\u07ff\u0800\uffff",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	560 [
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	561 u"",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	562 u"",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	563 u"", # First BOM has been read and skipped
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	564 u"",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	565 u"",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	566 u"\ufeff", # Second BOM has been read and emitted
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	567 u"\ufeff\x00", # "\x00" read and emitted
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	568 u"\ufeff\x00", # First byte of encoded u"\xff" read
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	569 u"\ufeff\x00\xff", # Second byte of encoded u"\xff" read
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	570 u"\ufeff\x00\xff", # First byte of encoded u"\u07ff" read
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	571 u"\ufeff\x00\xff\u07ff", # Second byte of encoded u"\u07ff" read
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	572 u"\ufeff\x00\xff\u07ff",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	573 u"\ufeff\x00\xff\u07ff",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	574 u"\ufeff\x00\xff\u07ff\u0800",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	575 u"\ufeff\x00\xff\u07ff\u0800",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	576 u"\ufeff\x00\xff\u07ff\u0800",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	577 u"\ufeff\x00\xff\u07ff\u0800\uffff",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	578 ]
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	579 )
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	580
40386 a3e0f1e11082 Change decode() so that it works with a buffer (i.e. unicode(..., 'utf-8-sig')) Walter Dörwald <walter@livinglogic.de> parents: 40284 diff changeset	581 def test_bug1601501(self):
a3e0f1e11082 Change decode() so that it works with a buffer (i.e. unicode(..., 'utf-8-sig')) Walter Dörwald <walter@livinglogic.de> parents: 40284 diff changeset	582 # SF bug #1601501: check that the codec works with a buffer
a3e0f1e11082 Change decode() so that it works with a buffer (i.e. unicode(..., 'utf-8-sig')) Walter Dörwald <walter@livinglogic.de> parents: 40284 diff changeset	583 unicode("\xef\xbb\xbf", "utf-8-sig")
a3e0f1e11082 Change decode() so that it works with a buffer (i.e. unicode(..., 'utf-8-sig')) Walter Dörwald <walter@livinglogic.de> parents: 40284 diff changeset	584
41445 8447314f500a Fix utf-8-sig incremental decoder, which didn't recognise a BOM when the Walter Dörwald <walter@livinglogic.de> parents: 40386 diff changeset	585 def test_bom(self):
8447314f500a Fix utf-8-sig incremental decoder, which didn't recognise a BOM when the Walter Dörwald <walter@livinglogic.de> parents: 40386 diff changeset	586 d = codecs.getincrementaldecoder("utf-8-sig")()
8447314f500a Fix utf-8-sig incremental decoder, which didn't recognise a BOM when the Walter Dörwald <walter@livinglogic.de> parents: 40386 diff changeset	587 s = u"spam"
8447314f500a Fix utf-8-sig incremental decoder, which didn't recognise a BOM when the Walter Dörwald <walter@livinglogic.de> parents: 40386 diff changeset	588 self.assertEqual(d.decode(s.encode("utf-8-sig")), s)
8447314f500a Fix utf-8-sig incremental decoder, which didn't recognise a BOM when the Walter Dörwald <walter@livinglogic.de> parents: 40386 diff changeset	589
43905 0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	590 def test_stream_bom(self):
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	591 unistring = u"ABC\u00A1\u2200XYZ"
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	592 bytestring = codecs.BOM_UTF8 + "ABC\xC2\xA1\xE2\x88\x80XYZ"
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	593
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	594 reader = codecs.getreader("utf-8-sig")
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	595 for sizehint in [None] + range(1, 11) + \
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	596 [64, 128, 256, 512, 1024]:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	597 istream = reader(StringIO.StringIO(bytestring))
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	598 ostream = StringIO.StringIO()
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	599 while 1:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	600 if sizehint is not None:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	601 data = istream.read(sizehint)
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	602 else:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	603 data = istream.read()
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	604
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	605 if not data:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	606 break
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	607 ostream.write(data)
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	608
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	609 got = ostream.getvalue()
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	610 self.assertEqual(got, unistring)
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	611
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	612 def test_stream_bare(self):
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	613 unistring = u"ABC\u00A1\u2200XYZ"
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	614 bytestring = "ABC\xC2\xA1\xE2\x88\x80XYZ"
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	615
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	616 reader = codecs.getreader("utf-8-sig")
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	617 for sizehint in [None] + range(1, 11) + \
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	618 [64, 128, 256, 512, 1024]:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	619 istream = reader(StringIO.StringIO(bytestring))
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	620 ostream = StringIO.StringIO()
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	621 while 1:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	622 if sizehint is not None:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	623 data = istream.read(sizehint)
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	624 else:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	625 data = istream.read()
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	626
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	627 if not data:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	628 break
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	629 ostream.write(data)
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	630
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	631 got = ostream.getvalue()
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	632 self.assertEqual(got, unistring)
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	633
25100 bc590beadf0b Check whether a string resize is necessary at the end Walter Dörwald <walter@livinglogic.de> parents: 24364 diff changeset	634 class EscapeDecodeTest(unittest.TestCase):
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	635 def test_empty(self):
25100 bc590beadf0b Check whether a string resize is necessary at the end Walter Dörwald <walter@livinglogic.de> parents: 24364 diff changeset	636 self.assertEquals(codecs.escape_decode(""), ("", 0))
bc590beadf0b Check whether a string resize is necessary at the end Walter Dörwald <walter@livinglogic.de> parents: 24364 diff changeset	637
27680 b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	638 class RecodingTest(unittest.TestCase):
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	639 def test_recoding(self):
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	640 f = StringIO.StringIO()
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	641 f2 = codecs.EncodedFile(f, "unicode_internal", "utf-8")
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	642 f2.write(u"a")
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	643 f2.close()
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	644 # Python used to crash on this at exit because of a refcount
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	645 # bug in _codecsmodule.c
20013 e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	646
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	647 # From RFC 3492
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	648 punycode_testcases = [
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	649 # A Arabic (Egyptian):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	650 (u"\u0644\u064A\u0647\u0645\u0627\u0628\u062A\u0643\u0644"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	651 u"\u0645\u0648\u0634\u0639\u0631\u0628\u064A\u061F",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	652 "egbpdaj6bu4bxfgehfvwxn"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	653 # B Chinese (simplified):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	654 (u"\u4ED6\u4EEC\u4E3A\u4EC0\u4E48\u4E0D\u8BF4\u4E2D\u6587",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	655 "ihqwcrb4cv8a8dqg056pqjye"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	656 # C Chinese (traditional):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	657 (u"\u4ED6\u5011\u7232\u4EC0\u9EBD\u4E0D\u8AAA\u4E2D\u6587",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	658 "ihqwctvzc91f659drss3x8bo0yb"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	659 # D Czech: Pro<ccaron>prost<ecaron>nemluv<iacute><ccaron>esky
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	660 (u"\u0050\u0072\u006F\u010D\u0070\u0072\u006F\u0073\u0074"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	661 u"\u011B\u006E\u0065\u006D\u006C\u0075\u0076\u00ED\u010D"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	662 u"\u0065\u0073\u006B\u0079",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	663 "Proprostnemluvesky-uyb24dma41a"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	664 # E Hebrew:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	665 (u"\u05DC\u05DE\u05D4\u05D4\u05DD\u05E4\u05E9\u05D5\u05D8"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	666 u"\u05DC\u05D0\u05DE\u05D3\u05D1\u05E8\u05D9\u05DD\u05E2"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	667 u"\u05D1\u05E8\u05D9\u05EA",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	668 "4dbcagdahymbxekheh6e0a7fei0b"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	669 # F Hindi (Devanagari):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	670 (u"\u092F\u0939\u0932\u094B\u0917\u0939\u093F\u0928\u094D"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	671 u"\u0926\u0940\u0915\u094D\u092F\u094B\u0902\u0928\u0939"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	672 u"\u0940\u0902\u092C\u094B\u0932\u0938\u0915\u0924\u0947"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	673 u"\u0939\u0948\u0902",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	674 "i1baa7eci9glrd9b2ae1bj0hfcgg6iyaf8o0a1dig0cd"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	675
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	676 #(G) Japanese (kanji and hiragana):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	677 (u"\u306A\u305C\u307F\u3093\u306A\u65E5\u672C\u8A9E\u3092"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	678 u"\u8A71\u3057\u3066\u304F\u308C\u306A\u3044\u306E\u304B",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	679 "n8jok5ay5dzabd5bym9f0cm5685rrjetr6pdxa"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	680
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	681 # (H) Korean (Hangul syllables):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	682 (u"\uC138\uACC4\uC758\uBAA8\uB4E0\uC0AC\uB78C\uB4E4\uC774"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	683 u"\uD55C\uAD6D\uC5B4\uB97C\uC774\uD574\uD55C\uB2E4\uBA74"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	684 u"\uC5BC\uB9C8\uB098\uC88B\uC744\uAE4C",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	685 "989aomsvi5e83db1d2a355cv1e0vak1dwrv93d5xbh15a0dt30a5j"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	686 "psd879ccm6fea98c"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	687
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	688 # (I) Russian (Cyrillic):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	689 (u"\u043F\u043E\u0447\u0435\u043C\u0443\u0436\u0435\u043E"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	690 u"\u043D\u0438\u043D\u0435\u0433\u043E\u0432\u043E\u0440"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	691 u"\u044F\u0442\u043F\u043E\u0440\u0443\u0441\u0441\u043A"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	692 u"\u0438",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	693 "b1abfaaepdrnnbgefbaDotcwatmq2g4l"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	694
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	695 # (J) Spanish: Porqu<eacute>nopuedensimplementehablarenEspa<ntilde>ol
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	696 (u"\u0050\u006F\u0072\u0071\u0075\u00E9\u006E\u006F\u0070"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	697 u"\u0075\u0065\u0064\u0065\u006E\u0073\u0069\u006D\u0070"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	698 u"\u006C\u0065\u006D\u0065\u006E\u0074\u0065\u0068\u0061"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	699 u"\u0062\u006C\u0061\u0072\u0065\u006E\u0045\u0073\u0070"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	700 u"\u0061\u00F1\u006F\u006C",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	701 "PorqunopuedensimplementehablarenEspaol-fmd56a"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	702
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	703 # (K) Vietnamese:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	704 # T<adotbelow>isaoh<odotbelow>kh<ocirc>ngth<ecirchookabove>ch\
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	705 # <ihookabove>n<oacute>iti<ecircacute>ngVi<ecircdotbelow>t
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	706 (u"\u0054\u1EA1\u0069\u0073\u0061\u006F\u0068\u1ECD\u006B"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	707 u"\u0068\u00F4\u006E\u0067\u0074\u0068\u1EC3\u0063\u0068"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	708 u"\u1EC9\u006E\u00F3\u0069\u0074\u0069\u1EBF\u006E\u0067"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	709 u"\u0056\u0069\u1EC7\u0074",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	710 "TisaohkhngthchnitingVit-kjcr8268qyxafd2f1b9g"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	711
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	712 #(L) 3<nen>B<gumi><kinpachi><sensei>
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	713 (u"\u0033\u5E74\u0042\u7D44\u91D1\u516B\u5148\u751F",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	714 "3B-ww4c5e180e575a65lsy2b"),
28774 c9bee11eaa8b Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 28678 diff changeset	715
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	716 # (M) <amuro><namie>-with-SUPER-MONKEYS
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	717 (u"\u5B89\u5BA4\u5948\u7F8E\u6075\u002D\u0077\u0069\u0074"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	718 u"\u0068\u002D\u0053\u0055\u0050\u0045\u0052\u002D\u004D"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	719 u"\u004F\u004E\u004B\u0045\u0059\u0053",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	720 "-with-SUPER-MONKEYS-pc58ag80a8qai00g7n9n"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	721
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	722 # (N) Hello-Another-Way-<sorezore><no><basho>
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	723 (u"\u0048\u0065\u006C\u006C\u006F\u002D\u0041\u006E\u006F"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	724 u"\u0074\u0068\u0065\u0072\u002D\u0057\u0061\u0079\u002D"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	725 u"\u305D\u308C\u305E\u308C\u306E\u5834\u6240",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	726 "Hello-Another-Way--fc4qua05auwb3674vfr0b"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	727
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	728 # (O) <hitotsu><yane><no><shita>2
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	729 (u"\u3072\u3068\u3064\u5C4B\u6839\u306E\u4E0B\u0032",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	730 "2-u9tlzr9756bt3uc0v"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	731
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	732 # (P) Maji<de>Koi<suru>5<byou><mae>
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	733 (u"\u004D\u0061\u006A\u0069\u3067\u004B\u006F\u0069\u3059"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	734 u"\u308B\u0035\u79D2\u524D",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	735 "MajiKoi5-783gue6qz075azm5e"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	736
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	737 # (Q) <pafii>de<runba>
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	738 (u"\u30D1\u30D5\u30A3\u30FC\u0064\u0065\u30EB\u30F3\u30D0",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	739 "de-jg4avhby1noc0d"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	740
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	741 # (R) <sono><supiido><de>
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	742 (u"\u305D\u306E\u30B9\u30D4\u30FC\u30C9\u3067",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	743 "d9juau41awczczp"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	744
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	745 # (S) -> $1.00 <-
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	746 (u"\u002D\u003E\u0020\u0024\u0031\u002E\u0030\u0030\u0020"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	747 u"\u003C\u002D",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	748 "-> $1.00 <--")
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	749 ]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	750
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	751 for i in punycode_testcases:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	752 if len(i)!=2:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	753 print repr(i)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	754
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	755 class PunycodeTest(unittest.TestCase):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	756 def test_encode(self):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	757 for uni, puny in punycode_testcases:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	758 # Need to convert both strings to lower case, since
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	759 # some of the extended encodings use upper case, but our
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	760 # code produces only lower case. Converting just puny to
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	761 # lower is also insufficient, since some of the input characters
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	762 # are upper case.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	763 self.assertEquals(uni.encode("punycode").lower(), puny.lower())
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	764
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	765 def test_decode(self):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	766 for uni, puny in punycode_testcases:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	767 self.assertEquals(uni, puny.decode("punycode"))
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	768
35484 4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	769 class UnicodeInternalTest(unittest.TestCase):
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	770 def test_bug1251300(self):
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	771 # Decoding with unicode_internal used to not correctly handle "code
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	772 # points" above 0x10ffff on UCS-4 builds.
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	773 if sys.maxunicode > 0xffff:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	774 ok = [
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	775 ("\x00\x10\xff\xff", u"\U0010ffff"),
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	776 ("\x00\x00\x01\x01", u"\U00000101"),
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	777 ("", u""),
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	778 ]
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	779 not_ok = [
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	780 "\x7f\xff\xff\xff",
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	781 "\x80\x00\x00\x00",
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	782 "\x81\x00\x00\x00",
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	783 "\x00",
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	784 "\x00\x00\x00\x00\x00",
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	785 ]
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	786 for internal, uni in ok:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	787 if sys.byteorder == "little":
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	788 internal = "".join(reversed(internal))
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	789 self.assertEquals(uni, internal.decode("unicode_internal"))
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	790 for internal in not_ok:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	791 if sys.byteorder == "little":
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	792 internal = "".join(reversed(internal))
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	793 self.assertRaises(UnicodeDecodeError, internal.decode,
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	794 "unicode_internal")
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	795
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	796 def test_decode_error_attributes(self):
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	797 if sys.maxunicode > 0xffff:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	798 try:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	799 "\x00\x00\x00\x00\x00\x11\x11\x00".decode("unicode_internal")
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	800 except UnicodeDecodeError, ex:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	801 self.assertEquals("unicode_internal", ex.encoding)
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	802 self.assertEquals("\x00\x00\x00\x00\x00\x11\x11\x00", ex.object)
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	803 self.assertEquals(4, ex.start)
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	804 self.assertEquals(8, ex.end)
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	805 else:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	806 self.fail()
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	807
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	808 def test_decode_callback(self):
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	809 if sys.maxunicode > 0xffff:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	810 codecs.register_error("UnicodeInternalTest", codecs.ignore_errors)
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	811 decoder = codecs.getdecoder("unicode_internal")
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	812 ab = u"ab".encode("unicode_internal")
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	813 ignored = decoder("%s\x22\x22\x22\x22%s" % (ab[:4], ab[4:]),
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	814 "UnicodeInternalTest")
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	815 self.assertEquals((u"ab", 12), ignored)
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	816
53855 7dca250343ea Issue 3739: The unicode-internal encoder now reports the number of characters Walter Dörwald <walter@livinglogic.de> parents: 43932 diff changeset	817 def test_encode_length(self):
7dca250343ea Issue 3739: The unicode-internal encoder now reports the number of characters Walter Dörwald <walter@livinglogic.de> parents: 43932 diff changeset	818 # Issue 3739
7dca250343ea Issue 3739: The unicode-internal encoder now reports the number of characters Walter Dörwald <walter@livinglogic.de> parents: 43932 diff changeset	819 encoder = codecs.getencoder("unicode_internal")
7dca250343ea Issue 3739: The unicode-internal encoder now reports the number of characters Walter Dörwald <walter@livinglogic.de> parents: 43932 diff changeset	820 self.assertEquals(encoder(u"a")[1], 1)
7dca250343ea Issue 3739: The unicode-internal encoder now reports the number of characters Walter Dörwald <walter@livinglogic.de> parents: 43932 diff changeset	821 self.assertEquals(encoder(u"\xe9\u0142")[1], 2)
7dca250343ea Issue 3739: The unicode-internal encoder now reports the number of characters Walter Dörwald <walter@livinglogic.de> parents: 43932 diff changeset	822
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	823 # From http://www.gnu.org/software/libidn/draft-josefsson-idn-test-vectors.html
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	824 nameprep_tests = [
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	825 # 3.1 Map to nothing.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	826 ('foo\xc2\xad\xcd\x8f\xe1\xa0\x86\xe1\xa0\x8bbar'
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	827 '\xe2\x80\x8b\xe2\x81\xa0baz\xef\xb8\x80\xef\xb8\x88\xef'
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	828 '\xb8\x8f\xef\xbb\xbf',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	829 'foobarbaz'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	830 # 3.2 Case folding ASCII U+0043 U+0041 U+0046 U+0045.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	831 ('CAFE',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	832 'cafe'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	833 # 3.3 Case folding 8bit U+00DF (german sharp s).
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	834 # The original test case is bogus; it says \xc3\xdf
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	835 ('\xc3\x9f',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	836 'ss'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	837 # 3.4 Case folding U+0130 (turkish capital I with dot).
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	838 ('\xc4\xb0',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	839 'i\xcc\x87'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	840 # 3.5 Case folding multibyte U+0143 U+037A.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	841 ('\xc5\x83\xcd\xba',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	842 '\xc5\x84 \xce\xb9'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	843 # 3.6 Case folding U+2121 U+33C6 U+1D7BB.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	844 # XXX: skip this as it fails in UCS-2 mode
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	845 #('\xe2\x84\xa1\xe3\x8f\x86\xf0\x9d\x9e\xbb',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	846 # 'telc\xe2\x88\x95kg\xcf\x83'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	847 (None, None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	848 # 3.7 Normalization of U+006a U+030c U+00A0 U+00AA.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	849 ('j\xcc\x8c\xc2\xa0\xc2\xaa',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	850 '\xc7\xb0 a'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	851 # 3.8 Case folding U+1FB7 and normalization.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	852 ('\xe1\xbe\xb7',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	853 '\xe1\xbe\xb6\xce\xb9'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	854 # 3.9 Self-reverting case folding U+01F0 and normalization.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	855 # The original test case is bogus, it says `\xc7\xf0'
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	856 ('\xc7\xb0',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	857 '\xc7\xb0'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	858 # 3.10 Self-reverting case folding U+0390 and normalization.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	859 ('\xce\x90',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	860 '\xce\x90'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	861 # 3.11 Self-reverting case folding U+03B0 and normalization.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	862 ('\xce\xb0',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	863 '\xce\xb0'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	864 # 3.12 Self-reverting case folding U+1E96 and normalization.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	865 ('\xe1\xba\x96',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	866 '\xe1\xba\x96'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	867 # 3.13 Self-reverting case folding U+1F56 and normalization.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	868 ('\xe1\xbd\x96',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	869 '\xe1\xbd\x96'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	870 # 3.14 ASCII space character U+0020.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	871 (' ',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	872 ' '),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	873 # 3.15 Non-ASCII 8bit space character U+00A0.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	874 ('\xc2\xa0',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	875 ' '),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	876 # 3.16 Non-ASCII multibyte space character U+1680.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	877 ('\xe1\x9a\x80',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	878 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	879 # 3.17 Non-ASCII multibyte space character U+2000.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	880 ('\xe2\x80\x80',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	881 ' '),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	882 # 3.18 Zero Width Space U+200b.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	883 ('\xe2\x80\x8b',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	884 ''),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	885 # 3.19 Non-ASCII multibyte space character U+3000.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	886 ('\xe3\x80\x80',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	887 ' '),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	888 # 3.20 ASCII control characters U+0010 U+007F.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	889 ('\x10\x7f',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	890 '\x10\x7f'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	891 # 3.21 Non-ASCII 8bit control character U+0085.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	892 ('\xc2\x85',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	893 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	894 # 3.22 Non-ASCII multibyte control character U+180E.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	895 ('\xe1\xa0\x8e',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	896 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	897 # 3.23 Zero Width No-Break Space U+FEFF.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	898 ('\xef\xbb\xbf',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	899 ''),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	900 # 3.24 Non-ASCII control character U+1D175.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	901 ('\xf0\x9d\x85\xb5',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	902 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	903 # 3.25 Plane 0 private use character U+F123.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	904 ('\xef\x84\xa3',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	905 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	906 # 3.26 Plane 15 private use character U+F1234.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	907 ('\xf3\xb1\x88\xb4',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	908 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	909 # 3.27 Plane 16 private use character U+10F234.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	910 ('\xf4\x8f\x88\xb4',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	911 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	912 # 3.28 Non-character code point U+8FFFE.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	913 ('\xf2\x8f\xbf\xbe',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	914 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	915 # 3.29 Non-character code point U+10FFFF.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	916 ('\xf4\x8f\xbf\xbf',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	917 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	918 # 3.30 Surrogate code U+DF42.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	919 ('\xed\xbd\x82',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	920 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	921 # 3.31 Non-plain text character U+FFFD.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	922 ('\xef\xbf\xbd',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	923 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	924 # 3.32 Ideographic description character U+2FF5.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	925 ('\xe2\xbf\xb5',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	926 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	927 # 3.33 Display property character U+0341.
28774 c9bee11eaa8b Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 28678 diff changeset	928 ('\xcd\x81',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	929 '\xcc\x81'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	930 # 3.34 Left-to-right mark U+200E.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	931 ('\xe2\x80\x8e',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	932 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	933 # 3.35 Deprecated U+202A.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	934 ('\xe2\x80\xaa',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	935 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	936 # 3.36 Language tagging character U+E0001.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	937 ('\xf3\xa0\x80\x81',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	938 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	939 # 3.37 Language tagging character U+E0042.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	940 ('\xf3\xa0\x81\x82',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	941 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	942 # 3.38 Bidi: RandALCat character U+05BE and LCat characters.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	943 ('foo\xd6\xbebar',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	944 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	945 # 3.39 Bidi: RandALCat character U+FD50 and LCat characters.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	946 ('foo\xef\xb5\x90bar',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	947 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	948 # 3.40 Bidi: RandALCat character U+FB38 and LCat characters.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	949 ('foo\xef\xb9\xb6bar',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	950 'foo \xd9\x8ebar'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	951 # 3.41 Bidi: RandALCat without trailing RandALCat U+0627 U+0031.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	952 ('\xd8\xa71',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	953 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	954 # 3.42 Bidi: RandALCat character U+0627 U+0031 U+0628.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	955 ('\xd8\xa71\xd8\xa8',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	956 '\xd8\xa71\xd8\xa8'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	957 # 3.43 Unassigned code point U+E0002.
28678 31cb8e0260db Skip nameprep test 3.43, as we do allow unassigned characters. The test Martin v. Löwis <martin@v.loewis.de> parents: 28671 diff changeset	958 # Skip this test as we allow unassigned
31cb8e0260db Skip nameprep test 3.43, as we do allow unassigned characters. The test Martin v. Löwis <martin@v.loewis.de> parents: 28671 diff changeset	959 #('\xf3\xa0\x80\x82',
31cb8e0260db Skip nameprep test 3.43, as we do allow unassigned characters. The test Martin v. Löwis <martin@v.loewis.de> parents: 28671 diff changeset	960 # None),
31cb8e0260db Skip nameprep test 3.43, as we do allow unassigned characters. The test Martin v. Löwis <martin@v.loewis.de> parents: 28671 diff changeset	961 (None, None),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	962 # 3.44 Larger test (shrinking).
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	963 # Original test case reads \xc3\xdf
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	964 ('X\xc2\xad\xc3\x9f\xc4\xb0\xe2\x84\xa1j\xcc\x8c\xc2\xa0\xc2'
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	965 '\xaa\xce\xb0\xe2\x80\x80',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	966 'xssi\xcc\x87tel\xc7\xb0 a\xce\xb0 '),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	967 # 3.45 Larger test (expanding).
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	968 # Original test case reads \xc3\x9f
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	969 ('X\xc3\x9f\xe3\x8c\x96\xc4\xb0\xe2\x84\xa1\xe2\x92\x9f\xe3\x8c'
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	970 '\x80',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	971 'xss\xe3\x82\xad\xe3\x83\xad\xe3\x83\xa1\xe3\x83\xbc\xe3'
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	972 '\x83\x88\xe3\x83\xabi\xcc\x87tel\x28d\x29\xe3\x82'
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	973 '\xa2\xe3\x83\x91\xe3\x83\xbc\xe3\x83\x88')
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	974 ]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	975
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	976
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	977 class NameprepTest(unittest.TestCase):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	978 def test_nameprep(self):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	979 from encodings.idna import nameprep
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	980 for pos, (orig, prepped) in enumerate(nameprep_tests):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	981 if orig is None:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	982 # Skipped
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	983 continue
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	984 # The Unicode strings are given in UTF-8
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	985 orig = unicode(orig, "utf-8")
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	986 if prepped is None:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	987 # Input contains prohibited characters
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	988 self.assertRaises(UnicodeError, nameprep, orig)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	989 else:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	990 prepped = unicode(prepped, "utf-8")
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	991 try:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	992 self.assertEquals(nameprep(orig), prepped)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	993 except Exception,e:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	994 raise test_support.TestFailed("Test 3.%d: %s" % (pos+1, str(e)))
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	995
37719 cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	996 class IDNACodecTest(unittest.TestCase):
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	997 def test_builtin_decode(self):
31795 bf37429f6d9f Add test case for unicode(somestring, "idna"). Martin v. Löwis <martin@v.loewis.de> parents: 28917 diff changeset	998 self.assertEquals(unicode("python.org", "idna"), u"python.org")
37719 cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	999 self.assertEquals(unicode("python.org.", "idna"), u"python.org.")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1000 self.assertEquals(unicode("xn--pythn-mua.org", "idna"), u"pyth\xf6n.org")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1001 self.assertEquals(unicode("xn--pythn-mua.org.", "idna"), u"pyth\xf6n.org.")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1002
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1003 def test_builtin_encode(self):
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1004 self.assertEquals(u"python.org".encode("idna"), "python.org")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1005 self.assertEquals("python.org.".encode("idna"), "python.org.")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1006 self.assertEquals(u"pyth\xf6n.org".encode("idna"), "xn--pythn-mua.org")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1007 self.assertEquals(u"pyth\xf6n.org.".encode("idna"), "xn--pythn-mua.org.")
31795 bf37429f6d9f Add test case for unicode(somestring, "idna"). Martin v. Löwis <martin@v.loewis.de> parents: 28917 diff changeset	1008
35435 9d343bc81863 Make IDNA return an empty string when the input is empty. Fixes #1163178. Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	1009 def test_stream(self):
9d343bc81863 Make IDNA return an empty string when the input is empty. Fixes #1163178. Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	1010 import StringIO
9d343bc81863 Make IDNA return an empty string when the input is empty. Fixes #1163178. Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	1011 r = codecs.getreader("idna")(StringIO.StringIO("abc"))
9d343bc81863 Make IDNA return an empty string when the input is empty. Fixes #1163178. Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	1012 r.read(3)
9d343bc81863 Make IDNA return an empty string when the input is empty. Fixes #1163178. Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	1013 self.assertEquals(r.read(), u"")
9d343bc81863 Make IDNA return an empty string when the input is empty. Fixes #1163178. Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	1014
37719 cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1015 def test_incremental_decode(self):
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1016 self.assertEquals(
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1017 "".join(codecs.iterdecode("python.org", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1018 u"python.org"
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1019 )
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1020 self.assertEquals(
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1021 "".join(codecs.iterdecode("python.org.", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1022 u"python.org."
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1023 )
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1024 self.assertEquals(
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1025 "".join(codecs.iterdecode("xn--pythn-mua.org.", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1026 u"pyth\xf6n.org."
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1027 )
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1028 self.assertEquals(
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1029 "".join(codecs.iterdecode("xn--pythn-mua.org.", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1030 u"pyth\xf6n.org."
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1031 )
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1032
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1033 decoder = codecs.getincrementaldecoder("idna")()
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1034 self.assertEquals(decoder.decode("xn--xam", ), u"")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1035 self.assertEquals(decoder.decode("ple-9ta.o", ), u"\xe4xample.")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1036 self.assertEquals(decoder.decode(u"rg"), u"")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1037 self.assertEquals(decoder.decode(u"", True), u"org")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1038
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1039 decoder.reset()
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1040 self.assertEquals(decoder.decode("xn--xam", ), u"")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1041 self.assertEquals(decoder.decode("ple-9ta.o", ), u"\xe4xample.")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1042 self.assertEquals(decoder.decode("rg."), u"org.")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1043 self.assertEquals(decoder.decode("", True), u"")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1044
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1045 def test_incremental_encode(self):
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1046 self.assertEquals(
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1047 "".join(codecs.iterencode(u"python.org", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1048 "python.org"
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1049 )
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1050 self.assertEquals(
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1051 "".join(codecs.iterencode(u"python.org.", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1052 "python.org."
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1053 )
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1054 self.assertEquals(
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1055 "".join(codecs.iterencode(u"pyth\xf6n.org.", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1056 "xn--pythn-mua.org."
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1057 )
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1058 self.assertEquals(
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1059 "".join(codecs.iterencode(u"pyth\xf6n.org.", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1060 "xn--pythn-mua.org."
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1061 )
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1062
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1063 encoder = codecs.getincrementalencoder("idna")()
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1064 self.assertEquals(encoder.encode(u"\xe4x"), "")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1065 self.assertEquals(encoder.encode(u"ample.org"), "xn--xample-9ta.")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1066 self.assertEquals(encoder.encode(u"", True), "org")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1067
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1068 encoder.reset()
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1069 self.assertEquals(encoder.encode(u"\xe4x"), "")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1070 self.assertEquals(encoder.encode(u"ample.org."), "xn--xample-9ta.org.")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1071 self.assertEquals(encoder.encode(u"", True), "")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1072
32572 8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1073 class CodecsModuleTest(unittest.TestCase):
8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1074
8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1075 def test_decode(self):
8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1076 self.assertEquals(codecs.decode('\xe4\xf6\xfc', 'latin-1'),
8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1077 u'\xe4\xf6\xfc')
33855 1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1078 self.assertRaises(TypeError, codecs.decode)
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1079 self.assertEquals(codecs.decode('abc'), u'abc')
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1080 self.assertRaises(UnicodeDecodeError, codecs.decode, '\xff', 'ascii')
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1081
32572 8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1082 def test_encode(self):
8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1083 self.assertEquals(codecs.encode(u'\xe4\xf6\xfc', 'latin-1'),
8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1084 '\xe4\xf6\xfc')
33855 1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1085 self.assertRaises(TypeError, codecs.encode)
35889 e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1086 self.assertRaises(LookupError, codecs.encode, "foo", "__spam__")
33855 1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1087 self.assertEquals(codecs.encode(u'abc'), 'abc')
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1088 self.assertRaises(UnicodeEncodeError, codecs.encode, u'\xffff', 'ascii')
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1089
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1090 def test_register(self):
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1091 self.assertRaises(TypeError, codecs.register)
35889 e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1092 self.assertRaises(TypeError, codecs.register, 42)
33855 1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1093
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1094 def test_lookup(self):
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1095 self.assertRaises(TypeError, codecs.lookup)
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1096 self.assertRaises(LookupError, codecs.lookup, "__spam__")
35889 e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1097 self.assertRaises(LookupError, codecs.lookup, " ")
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1098
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1099 def test_getencoder(self):
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1100 self.assertRaises(TypeError, codecs.getencoder)
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1101 self.assertRaises(LookupError, codecs.getencoder, "__spam__")
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1102
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1103 def test_getdecoder(self):
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1104 self.assertRaises(TypeError, codecs.getdecoder)
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1105 self.assertRaises(LookupError, codecs.getdecoder, "__spam__")
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1106
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1107 def test_getreader(self):
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1108 self.assertRaises(TypeError, codecs.getreader)
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1109 self.assertRaises(LookupError, codecs.getreader, "__spam__")
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1110
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1111 def test_getwriter(self):
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1112 self.assertRaises(TypeError, codecs.getwriter)
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1113 self.assertRaises(LookupError, codecs.getwriter, "__spam__")
32572 8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1114
33782 a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1115 class StreamReaderTest(unittest.TestCase):
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1116
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1117 def setUp(self):
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1118 self.reader = codecs.getreader('utf-8')
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1119 self.stream = StringIO.StringIO('\xed\x95\x9c\n\xea\xb8\x80')
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1120
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1121 def test_readlines(self):
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1122 f = self.reader(self.stream)
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1123 self.assertEquals(f.readlines(), [u'\ud55c\n', u'\uae00'])
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1124
40239 7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1125 class EncodedFileTest(unittest.TestCase):
40284 ff678669aa6e Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 40274 diff changeset	1126
40239 7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1127 def test_basic(self):
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1128 f = StringIO.StringIO('\xed\x95\x9c\n\xea\xb8\x80')
40249 9663100c95ca Fix the new EncodedFile test to work with big endian platforms. Georg Brandl <georg@python.org> parents: 40239 diff changeset	1129 ef = codecs.EncodedFile(f, 'utf-16-le', 'utf-8')
9663100c95ca Fix the new EncodedFile test to work with big endian platforms. Georg Brandl <georg@python.org> parents: 40239 diff changeset	1130 self.assertEquals(ef.read(), '\\\xd5\n\x00\x00\xae')
40239 7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1131
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1132 f = StringIO.StringIO()
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1133 ef = codecs.EncodedFile(f, 'utf-8', 'latin1')
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1134 ef.write('\xc3\xbc')
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1135 self.assertEquals(f.getvalue(), '\xfc')
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1136
35269 2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1137 class Str2StrTest(unittest.TestCase):
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1138
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1139 def test_read(self):
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1140 sin = "\x80".encode("base64_codec")
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1141 reader = codecs.getreader("base64_codec")(StringIO.StringIO(sin))
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1142 sout = reader.read()
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1143 self.assertEqual(sout, "\x80")
58472 6ffcdafa0118 use assert[Not]IsInstance where appropriate Ezio Melotti <ezio.melotti@gmail.com> parents: 56033 diff changeset	1144 self.assertIsInstance(sout, str)
35269 2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1145
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1146 def test_readline(self):
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1147 sin = "\x80".encode("base64_codec")
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1148 reader = codecs.getreader("base64_codec")(StringIO.StringIO(sin))
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1149 sout = reader.readline()
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1150 self.assertEqual(sout, "\x80")
58472 6ffcdafa0118 use assert[Not]IsInstance where appropriate Ezio Melotti <ezio.melotti@gmail.com> parents: 56033 diff changeset	1151 self.assertIsInstance(sout, str)
35269 2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1152
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1153 all_unicode_encodings = [
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1154 "ascii",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1155 "base64_codec",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1156 "big5",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1157 "big5hkscs",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1158 "charmap",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1159 "cp037",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1160 "cp1006",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1161 "cp1026",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1162 "cp1140",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1163 "cp1250",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1164 "cp1251",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1165 "cp1252",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1166 "cp1253",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1167 "cp1254",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1168 "cp1255",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1169 "cp1256",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1170 "cp1257",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1171 "cp1258",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1172 "cp424",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1173 "cp437",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1174 "cp500",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1175 "cp737",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1176 "cp775",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1177 "cp850",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1178 "cp852",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1179 "cp855",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1180 "cp856",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1181 "cp857",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1182 "cp860",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1183 "cp861",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1184 "cp862",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1185 "cp863",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1186 "cp864",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1187 "cp865",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1188 "cp866",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1189 "cp869",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1190 "cp874",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1191 "cp875",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1192 "cp932",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1193 "cp949",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1194 "cp950",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1195 "euc_jis_2004",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1196 "euc_jisx0213",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1197 "euc_jp",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1198 "euc_kr",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1199 "gb18030",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1200 "gb2312",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1201 "gbk",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1202 "hex_codec",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1203 "hp_roman8",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1204 "hz",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1205 "idna",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1206 "iso2022_jp",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1207 "iso2022_jp_1",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1208 "iso2022_jp_2",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1209 "iso2022_jp_2004",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1210 "iso2022_jp_3",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1211 "iso2022_jp_ext",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1212 "iso2022_kr",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1213 "iso8859_1",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1214 "iso8859_10",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1215 "iso8859_11",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1216 "iso8859_13",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1217 "iso8859_14",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1218 "iso8859_15",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1219 "iso8859_16",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1220 "iso8859_2",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1221 "iso8859_3",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1222 "iso8859_4",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1223 "iso8859_5",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1224 "iso8859_6",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1225 "iso8859_7",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1226 "iso8859_8",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1227 "iso8859_9",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1228 "johab",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1229 "koi8_r",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1230 "koi8_u",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1231 "latin_1",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1232 "mac_cyrillic",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1233 "mac_greek",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1234 "mac_iceland",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1235 "mac_latin2",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1236 "mac_roman",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1237 "mac_turkish",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1238 "palmos",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1239 "ptcp154",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1240 "punycode",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1241 "raw_unicode_escape",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1242 "rot_13",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1243 "shift_jis",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1244 "shift_jis_2004",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1245 "shift_jisx0213",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1246 "tis_620",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1247 "unicode_escape",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1248 "unicode_internal",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1249 "utf_16",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1250 "utf_16_be",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1251 "utf_16_le",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1252 "utf_7",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1253 "utf_8",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1254 ]
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1255
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1256 if hasattr(codecs, "mbcs_encode"):
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1257 all_unicode_encodings.append("mbcs")
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1258
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1259 # The following encodings work only with str, not unicode
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1260 all_string_encodings = [
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1261 "quopri_codec",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1262 "string_escape",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1263 "uu_codec",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1264 ]
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1265
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1266 # The following encoding is not tested, because it's not supposed
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1267 # to work:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1268 # "undefined"
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1269
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1270 # The following encodings don't work in stateful mode
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1271 broken_unicode_with_streams = [
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1272 "base64_codec",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1273 "hex_codec",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1274 "punycode",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1275 "unicode_internal"
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1276 ]
40251 e16800ac1eea Bug #1586613: fix zlib and bz2 codecs' incremental en/decoders. Georg Brandl <georg@python.org> parents: 40249 diff changeset	1277 broken_incremental_coders = broken_unicode_with_streams[:]
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1278
40271 c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1279 # The following encodings only support "strict" mode
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1280 only_strict_mode = [
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1281 "idna",
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1282 "zlib_codec",
40274 bc8ffc3e3cdb I'm assuming this is correct, it fixes the tests so they pass again Neal Norwitz <nnorwitz@gmail.com> parents: 40271 diff changeset	1283 "bz2_codec",
40271 c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1284 ]
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1285
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1286 try:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1287 import bz2
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1288 except ImportError:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1289 pass
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1290 else:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1291 all_unicode_encodings.append("bz2_codec")
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1292 broken_unicode_with_streams.append("bz2_codec")
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1293
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1294 try:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1295 import zlib
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1296 except ImportError:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1297 pass
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1298 else:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1299 all_unicode_encodings.append("zlib_codec")
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1300 broken_unicode_with_streams.append("zlib_codec")
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1301
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1302 class BasicUnicodeTest(unittest.TestCase):
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1303 def test_basics(self):
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1304 s = u"abc123" # all codecs should be able to encode these
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1305 for encoding in all_unicode_encodings:
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1306 name = codecs.lookup(encoding).name
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1307 if encoding.endswith("_codec"):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1308 name += "_codec"
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1309 elif encoding == "latin_1":
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1310 name = "latin_1"
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1311 self.assertEqual(encoding.replace("_", "-"), name.replace("_", "-"))
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1312 (bytes, size) = codecs.getencoder(encoding)(s)
53855 7dca250343ea Issue 3739: The unicode-internal encoder now reports the number of characters Walter Dörwald <walter@livinglogic.de> parents: 43932 diff changeset	1313 self.assertEqual(size, len(s), "%r != %r (encoding=%r)" % (size, len(s), encoding))
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1314 (chars, size) = codecs.getdecoder(encoding)(bytes)
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1315 self.assertEqual(chars, s, "%r != %r (encoding=%r)" % (chars, s, encoding))
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1316
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1317 if encoding not in broken_unicode_with_streams:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1318 # check stream reader/writer
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1319 q = Queue()
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1320 writer = codecs.getwriter(encoding)(q)
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1321 encodedresult = ""
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1322 for c in s:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1323 writer.write(c)
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1324 encodedresult += q.read()
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1325 q = Queue()
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1326 reader = codecs.getreader(encoding)(q)
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1327 decodedresult = u""
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1328 for c in encodedresult:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1329 q.write(c)
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1330 decodedresult += reader.read()
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1331 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1332
40251 e16800ac1eea Bug #1586613: fix zlib and bz2 codecs' incremental en/decoders. Georg Brandl <georg@python.org> parents: 40249 diff changeset	1333 if encoding not in broken_incremental_coders:
37077 38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1334 # check incremental decoder/encoder (fetched via the Python
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1335 # and C API) and iterencode()/iterdecode()
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1336 try:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1337 encoder = codecs.getincrementalencoder(encoding)()
37077 38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1338 cencoder = _testcapi.codec_incrementalencoder(encoding)
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1339 except LookupError: # no IncrementalEncoder
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1340 pass
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1341 else:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1342 # check incremental decoder/encoder
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1343 encodedresult = ""
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1344 for c in s:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1345 encodedresult += encoder.encode(c)
37708 230fff6e0618 Call encode()/decode() with final==True as the last call in the Walter Dörwald <walter@livinglogic.de> parents: 37077 diff changeset	1346 encodedresult += encoder.encode(u"", True)
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1347 decoder = codecs.getincrementaldecoder(encoding)()
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1348 decodedresult = u""
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1349 for c in encodedresult:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1350 decodedresult += decoder.decode(c)
37708 230fff6e0618 Call encode()/decode() with final==True as the last call in the Walter Dörwald <walter@livinglogic.de> parents: 37077 diff changeset	1351 decodedresult += decoder.decode("", True)
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1352 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1353
37077 38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1354 # check C API
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1355 encodedresult = ""
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1356 for c in s:
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1357 encodedresult += cencoder.encode(c)
37708 230fff6e0618 Call encode()/decode() with final==True as the last call in the Walter Dörwald <walter@livinglogic.de> parents: 37077 diff changeset	1358 encodedresult += cencoder.encode(u"", True)
37077 38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1359 cdecoder = _testcapi.codec_incrementaldecoder(encoding)
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1360 decodedresult = u""
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1361 for c in encodedresult:
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1362 decodedresult += cdecoder.decode(c)
37708 230fff6e0618 Call encode()/decode() with final==True as the last call in the Walter Dörwald <walter@livinglogic.de> parents: 37077 diff changeset	1363 decodedresult += cdecoder.decode("", True)
37077 38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1364 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1365
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1366 # check iterencode()/iterdecode()
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1367 result = u"".join(codecs.iterdecode(codecs.iterencode(s, encoding), encoding))
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1368 self.assertEqual(result, s, "%r != %r (encoding=%r)" % (result, s, encoding))
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1369
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1370 # check iterencode()/iterdecode() with empty string
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1371 result = u"".join(codecs.iterdecode(codecs.iterencode(u"", encoding), encoding))
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1372 self.assertEqual(result, u"")
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1373
40271 c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1374 if encoding not in only_strict_mode:
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1375 # check incremental decoder/encoder with errors argument
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1376 try:
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1377 encoder = codecs.getincrementalencoder(encoding)("ignore")
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1378 cencoder = _testcapi.codec_incrementalencoder(encoding, "ignore")
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1379 except LookupError: # no IncrementalEncoder
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1380 pass
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1381 else:
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1382 encodedresult = "".join(encoder.encode(c) for c in s)
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1383 decoder = codecs.getincrementaldecoder(encoding)("ignore")
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1384 decodedresult = u"".join(decoder.decode(c) for c in encodedresult)
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1385 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
40284 ff678669aa6e Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 40274 diff changeset	1386
40271 c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1387 encodedresult = "".join(cencoder.encode(c) for c in s)
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1388 cdecoder = _testcapi.codec_incrementaldecoder(encoding, "ignore")
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1389 decodedresult = u"".join(cdecoder.decode(c) for c in encodedresult)
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1390 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1391
34750 679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1392 def test_seek(self):
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1393 # all codecs should be able to encode these
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1394 s = u"%s\n%s\n" % (100u"abc123", 100u"def456")
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1395 for encoding in all_unicode_encodings:
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1396 if encoding == "idna": # FIXME: See SF bug #1163178
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1397 continue
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1398 if encoding in broken_unicode_with_streams:
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1399 continue
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1400 reader = codecs.getreader(encoding)(StringIO.StringIO(s.encode(encoding)))
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1401 for t in xrange(5):
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1402 # Test that calling seek resets the internal codec state and buffers
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1403 reader.seek(0, 0)
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1404 line = reader.readline()
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1405 self.assertEqual(s[:len(line)], line)
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1406
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1407 def test_bad_decode_args(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1408 for encoding in all_unicode_encodings:
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1409 decoder = codecs.getdecoder(encoding)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1410 self.assertRaises(TypeError, decoder)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1411 if encoding not in ("idna", "punycode"):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1412 self.assertRaises(TypeError, decoder, 42)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1413
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1414 def test_bad_encode_args(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1415 for encoding in all_unicode_encodings:
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1416 encoder = codecs.getencoder(encoding)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1417 self.assertRaises(TypeError, encoder)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1418
38787 70b166ca1c49 Verify the crash due to EncodingMap not initialized does not return Neal Norwitz <nnorwitz@gmail.com> parents: 37719 diff changeset	1419 def test_encoding_map_type_initialized(self):
70b166ca1c49 Verify the crash due to EncodingMap not initialized does not return Neal Norwitz <nnorwitz@gmail.com> parents: 37719 diff changeset	1420 from encodings import cp1140
70b166ca1c49 Verify the crash due to EncodingMap not initialized does not return Neal Norwitz <nnorwitz@gmail.com> parents: 37719 diff changeset	1421 # This used to crash, we are only verifying there's no crash.
70b166ca1c49 Verify the crash due to EncodingMap not initialized does not return Neal Norwitz <nnorwitz@gmail.com> parents: 37719 diff changeset	1422 table_type = type(cp1140.encoding_table)
70b166ca1c49 Verify the crash due to EncodingMap not initialized does not return Neal Norwitz <nnorwitz@gmail.com> parents: 37719 diff changeset	1423 self.assertEqual(table_type, table_type)
70b166ca1c49 Verify the crash due to EncodingMap not initialized does not return Neal Norwitz <nnorwitz@gmail.com> parents: 37719 diff changeset	1424
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1425 class BasicStrTest(unittest.TestCase):
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1426 def test_basics(self):
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1427 s = "abc123"
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1428 for encoding in all_string_encodings:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1429 (bytes, size) = codecs.getencoder(encoding)(s)
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1430 self.assertEqual(size, len(s))
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1431 (chars, size) = codecs.getdecoder(encoding)(bytes)
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1432 self.assertEqual(chars, s, "%r != %r (encoding=%r)" % (chars, s, encoding))
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1433
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1434 class CharmapTest(unittest.TestCase):
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1435 def test_decode_with_string_map(self):
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1436 self.assertEquals(
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1437 codecs.charmap_decode("\x00\x01\x02", "strict", u"abc"),
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1438 (u"abc", 3)
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1439 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1440
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1441 self.assertEquals(
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1442 codecs.charmap_decode("\x00\x01\x02", "replace", u"ab"),
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1443 (u"ab\ufffd", 3)
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1444 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1445
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1446 self.assertEquals(
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1447 codecs.charmap_decode("\x00\x01\x02", "replace", u"ab\ufffe"),
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1448 (u"ab\ufffd", 3)
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1449 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1450
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1451 self.assertEquals(
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1452 codecs.charmap_decode("\x00\x01\x02", "ignore", u"ab"),
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1453 (u"ab", 3)
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1454 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1455
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1456 self.assertEquals(
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1457 codecs.charmap_decode("\x00\x01\x02", "ignore", u"ab\ufffe"),
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1458 (u"ab", 3)
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1459 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1460
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1461 allbytes = "".join(chr(i) for i in xrange(256))
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1462 self.assertEquals(
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1463 codecs.charmap_decode(allbytes, "ignore", u""),
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1464 (u"", len(allbytes))
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1465 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1466
40239 7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1467 class WithStmtTest(unittest.TestCase):
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1468 def test_encodedfile(self):
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1469 f = StringIO.StringIO("\xc3\xbc")
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1470 with codecs.EncodedFile(f, "latin-1", "utf-8") as ef:
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1471 self.assertEquals(ef.read(), "\xfc")
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1472
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1473 def test_streamreaderwriter(self):
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1474 f = StringIO.StringIO("\xc3\xbc")
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1475 info = codecs.lookup("utf-8")
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1476 with codecs.StreamReaderWriter(f, info.streamreader,
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1477 info.streamwriter, 'strict') as srw:
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1478 self.assertEquals(srw.read(), u"\xfc")
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1479
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1480
20013 e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	1481 def test_main():
28917 1741af48fd9f Combine the functionality of test_support.run_unittest() Walter Dörwald <walter@livinglogic.de> parents: 28774 diff changeset	1482 test_support.run_unittest(
42649 c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	1483 UTF32Test,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	1484 UTF32LETest,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	1485 UTF32BETest,
28917 1741af48fd9f Combine the functionality of test_support.run_unittest() Walter Dörwald <walter@livinglogic.de> parents: 28774 diff changeset	1486 UTF16Test,
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	1487 UTF16LETest,
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	1488 UTF16BETest,
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	1489 UTF8Test,
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	1490 UTF8SigTest,
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1491 UTF7Test,
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1492 UTF16ExTest,
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1493 ReadBufferTest,
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1494 CharBufferTest,
28917 1741af48fd9f Combine the functionality of test_support.run_unittest() Walter Dörwald <walter@livinglogic.de> parents: 28774 diff changeset	1495 EscapeDecodeTest,
1741af48fd9f Combine the functionality of test_support.run_unittest() Walter Dörwald <walter@livinglogic.de> parents: 28774 diff changeset	1496 RecodingTest,
1741af48fd9f Combine the functionality of test_support.run_unittest() Walter Dörwald <walter@livinglogic.de> parents: 28774 diff changeset	1497 PunycodeTest,
35484 4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	1498 UnicodeInternalTest,
31795 bf37429f6d9f Add test case for unicode(somestring, "idna"). Martin v. Löwis <martin@v.loewis.de> parents: 28917 diff changeset	1499 NameprepTest,
37719 cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1500 IDNACodecTest,
33782 a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1501 CodecsModuleTest,
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1502 StreamReaderTest,
40239 7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1503 EncodedFileTest,
35269 2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1504 Str2StrTest,
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1505 BasicUnicodeTest,
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1506 BasicStrTest,
40239 7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1507 CharmapTest,
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1508 WithStmtTest,
28917 1741af48fd9f Combine the functionality of test_support.run_unittest() Walter Dörwald <walter@livinglogic.de> parents: 28774 diff changeset	1509 )
20013 e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	1510
e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	1511
e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	1512 if __name__ == "__main__":
e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	1513 test_main()

Mercurial > cpython-lru_cache-weakref

annotate Lib/test/test_codecs.py @ 58765:2ba7677bfa95 legacy-trunk